How does √(-3cos(t)/2)^2 +(3sin(t)/2)^2 (3cos(t)/2 - 9/2)^2 equal 3/4√4sin^2(t)-12cos(t)+17 instead of √cos^2(t) -254cos(t)/8 +81/4

Which one is correct or they're both wrong?

what is with all these √(...)^2 ??

√x^2 = |x|
Heck. Just plug in x=0 and see what happens
√(-3cos(0)/2)^2 +(3sin(0)/2)^2 (3cos(0)/2 - 9/2)^2 = 3/2
3/4√4sin^2(0)-12cos(0)+17 = 5
√cos^2(0) -254cos(0)/8 +81/4 = -21/2

So, clearly you need some parentheses to make things work.
Go to wolframalpha and type in your expressions. It will display what it thinks you mean. Use parentheses to make it clear what you want.