By using the formula cos (A+B) = cos A cos B - sin A sin B , prove that cos A - cos 3A = 4 sin^2 A cos A

well, how far did you get?

cos3A = cos(2A+A) = cos2A cosA - sin2A sinA
= (1-2Sin^2A)(cosA) - (2SinAcosA)(sinA)
try and finish it up from here