Im trying to figure out how to find the equaton for the brokerage fee. Ive looked everywhere in my math book but i cant figure it out. Please help.

Brokerage fees are usually not a formula or equation, but a table of prices. Fees vary by brokerage houses. I suggest you go to web site for E-Trade and/or Schwab and look at their fee schedules