Factor the difference of two cubes:

5x^4y^2-40xy^5

a^3 - b^3 = (a-b)(a^2+ab+b^2)

5x^4y^2 - 40xy^5 = 5xy^2(x^3 - 8y^3)
= 5xy^2(x^3 - (2y)^3))
= 5xy^2(x-2y)(x^2 + 2xy+4y^2)