How to prove tan4x = 1-8 sin squared of cos squared of x

I don't think you really meant

tan(4x) = 1 - sin^2(cos^2(x)), so do you mean

tan 4x = 1 - 8 sin^2 x cos^2 x?

Let's see.
8sin^2 x cos^2 x = 2*sin^2(2x))
tan(4x) =? 1 - 2sin^2(2x)

Naah. Check your problem again.

However, cos(4x) = 1 - 2sin^2(2x))