Using the "completing the square" technique, the maximum value of the function f(x)=sin^2(2x)-2cos^2x is

f(x) = sin^2(2x)-2cos^2x

= (2 sinx cosx)^2-2cos^2x
= 4 sin^2x cos^2x - 2cos^2x
= (4 - 4cos^2x)cos^2x - 2cos^2x
= -4cos^4x + 4cos^2x - 2cos^2x
= -4cos^4x + 2cos^2x
= -4(cos^4x - 1/2 cos^2x)
= -4(cos^4x - 1/2 cos^2x + 1/4) + 1
= -4(cos^2x - 1/4)^2 + 1/4
so the maximum is where cos^2x = 1/2
that is, at x = π/3
and f(π/3) = 1/4