Use the quotient rule to find the derivative of cot x.

I tried to solve this, but somehow I got (sec^2x)/(tanx)^2.

this is (almost) correct, since

sec^2/tan^2 = 1/cos^2 * cos^2/sin^2 = csc^2

Actually, the derivative is -csc^2x

cotx = cosx/sinx
d/dx(cotx) = (-sinx*sinx - cosx*cosx)/sin^2x
= -(sin^2x-cos^2x)/sin^2x
= -1/sin^2x
= -csc^2x

Next time, show your work, so we can see what went wrong.