My homework is to solve by completing the square. But how can you complete the square on this problem?

-3x^2 + 7x - 8

The -3 doesn't factor out.

Is the expression you just wrote supposed to be equal to zero? You cannot solve for x unless you write an equation.

Write it as -3[x^2 - (7/3)x + 8/3]= 0
This implies [x^2 - (7/3)x + 8/3]= 0
For a perfect square within the brackets, you'd want to rewrite it as
[x-(7/6)]^2 + 8/3 - 49/36
= [x-(7/6)]^2 + 47/36 = 0