what's the derivative of cos3x

The derivative of cos(3x) is -3sin(3x)

You need to learn the "chain rule" to understand how that was obtained.

Let u = 3x
The derivative of cos u with respect to x is
d(cosu)/du * du/dx = -sin u * 3
= -3 sin(3x)

If what you want is the derivative of cos^3 x, that is a different problem.
The answer to that is 3 cos^2x * (-sinx)