Does (1-cosx)^2 simplify to sin^2(x)?

no

you are thinking of 1 - cos^2 x , which is equal to sin^2 x

in yours, notice where the exponent is,
it is outside the brackets, so you are squaring a binomial and you would get
1 - 2cosx + cos^2 x