x^2+9y^2-25m^2-16n^2+6xy+40mn

Rearrange:

(x^2 + 6xy + 9y^2) - (25m^2 -40mn + 16n^2)
Factor as perfect squares:
= (x +3y)^2 - (5m - 4n)^2
Factor again as difference of squares:
= (x + 3y + 5m - 4n)(x + 3y - 5m + 4n)