From the method Reiny has supplied, which applies to equal payments for 30 years, where r=5%/12=0.0041666...
Monthly Payment = $805.23
The rule of 78 assumes the borrower owes all the interest at the start, so together with the principal, the borrower owes:
meaning the finance charges are
$289883.68-150000=139883.68 over 360 payments.
The finance charge (fixed interest charge) is subdivided into 360 unit for the first month, 359 units for the second, .... 1 unit for the last month for a total of 64980 units.
So if the loan is prepaid after 10 years, the "refund" of the interest is the number of units left for the next 20 years, 1+2+...+240=28920 units, or
So the borrower will have to pay, after 10 years of monthly payments of 805.23, the remaining principal of 100000 + remaining interest of 139883.68-62256.63
=77627.05 for a total of 177627.05 to pay off the mortgage.
So the borrower will have paid
for the initial borrowed sum of 150000 (instead of 289883.68 if he had paid over 30 years).
This method of calculation was to simplify calculations before calculators and computers were available, and benefits the lender by penalizing pre-payments, and also making the borrower owe all the interest (called finance charges) as soon as the load is made. According to Wikipedia, this method of calculation is prohibited in the US for mortgages and loans over 61 months.
The unpaid principal balance is a fair method by which all payments are applied to the current interest, and the remainder to reduce the principal.
The amount remaining to pay off the mortgage after 10 years would be (r=0.004166...)
This is considerably less than the rule of 78.