(xsquare-5x)square-36

(x^2-5x)^2-36

= (x^2-5x)^2 - 6^2
recall that a^2-b^2 = (a+b)(a-b). so, now you have
= (x^2-5x+6)(x^2-5x-6)
= (x-3)(x-2)(x-6)(x+1)