if x = ucos(pi/4) - vsin(pi/4) and y = usin(pi/4) + vcos(pi/4), find A, B and C so that,

5x^2 - 6xy + 5y^2 = Au^2 + Buv + Cv^2.

Thanks, I really don't know what to do, I've already used almost 20 sheets of paper. Please help me. Thanks.

Wow, 20 sheets of paper ?

Since you titled your post "College Algebra & Trig"
I will assume you know that both cos(π/4) and sin(π/4) = √2/2
so x = (√2/2)u - (√2/2)v and
y = (√2/2)u + (√2/2)v

In 5x^2 - 6xy + 5y^2 we will need:

( (√2/2)u - (√2/2)v )^2
( (√2/2)u + (√2/2)v )^2 and
( (√2/2)u - (√2/2)v)*( (√2/2)u + (√2/2)v )

let's do those again of time:
( (√2/2)u - (√2/2)v )^2
= (1/2)u^2 - uv + 1/2 v^2
= (1/2)(u^2 - 2uv + v^2)
= (1/2)(u-v)^2 ----------> x^2

( (√2/2)u + (√2/2)v )^2
= (1/2)(u^2 + 2uv + v^2)
= (1/2)(u+v)^2 ----------> y^2

( (√2/2)u - (√2/2)v)*( (√2/2)u + (√2/2)v )
= (1/2)u^2 - (1/2)v^2
= (1/2)(u^2 - v^2) ---------> xy

so 5x^2 -6xy + 5y^2
= (5/2)(u-v)^2 - 3(u^2 - y^2) + (5/2)(u+v)^2
= (5/2)(u^2 - 2uv + v^2) - 3(u^2-v^2) + (5/2)(u^2 + 2uv + v^2)
= (1/2)( 5u^2 - 10uv + 5v^2 - 6u^2 + 6v^2 + 5u^2 + 10uv + 5v^2
= (1/2)(4u^2 + 16v^2
= 2u^2 + 8v^2

Compare that to Au^2 + Buv + Cv^2
we conclude that A = 2 , B = 0 and C = 8
So A+B+C = 2+0+8 = 10

Please check my arithmetic, I did not write it out first, and it is easy to make a typo here.
Perhaps somebody else will do this using complex numbers and De Moivre's Theorem, and get a more elegant solution. I noticed that half way through my method, but stubbornly continued this way.