rotate the axes and eliminate the xy term of 3(x^2)+2xy+3(y^2)-8

you must have come across the following:

for ax^2 + 2hxy + b^2 + 2gx + 2fy + c = 0

then the xy term can be eliminated by a rotation through an angle of Ø, where
tan 2Ø = 2h/(b-a), a ≠ b
if a = b, then Ø = 45°

since 45 = cos 45 = 1/√2

the rotation matrix would be
1/√2 1/√2
-1/√2 1/√2