Rewrite 6x^2 - sqrt(3)xy + 5y^2 - 9 = 0 with the xy term eliminated.
the graph is rotated through an angle θ where
cot2θ = (A-C)/B = -1/√3 so θ = -π/6
So, we need new x'-y' axes where
x = (x'(√3/2)-y'(-1/2))
y = (x'(-1/2)+y'(√3/2))
expand the expressions (yes, you need to pay careful attention to the details!) and you end up with
13x'^2 + 9y'^2 = 18