How do I factor

a^2-10ab+3b^2?

Treat it as if it were a quadratic equation with

a = x as the independent variable
1 = a' as the coefficient of that variable
-10b = b' the coefficent of x
3b^2 = c = the constant
The root values of a are, from the quadratic equation solution formula,
= (1/2)[10b +/- sqrt(100b^2 -12 b^2)]
= (1/2)[10 b +/- sqrt(88b^2)
= (1/2)[10b +/- 2 sqrt 22b^2]
= [5 +/- sqrt(22)] b
= 9.69 b or 0.3096 b
The factors are therefore
(a - 9.6904 b)(a - 0.3096 b)

If this seems confusing, you could have gotten the same result by completing the square.