factor out the greatest common factor -36m^2n^3-60mn^4+18m^4n^2

Try factoring out 6mn^2.

-36m^2n^3-60mn^4+18m^4n^2 =
6mn^2(-6mn -10n^2 +3m^3)