x+2a is a factor of (x^3+2(a-2b)x^2-b(8a-3b)x+6ab^2)

I assume you want to find a and b so it divides evenly?

x^3 + 2(a-2b)x^2 - b(8a-3b)x + 6ab^2
-------------------------------------
x+2a

= x^2 - 4bx + 3b^2 remainder 6ab(1-b)

so, to divide evenly, either a=0 or b=1

check:

a=0: x^3 - 4bx^2 + 3b^2x + 0 = x(x^2-4bx+3b^2) for any b

b=1: x^3 + 2(a-2)x^2 - (8a+3)x + 6a
= (x+2a)(x^2 - 4x - 3) for any a