ac^2-a^3-a^2b+ab^2+b^2-bc^2

I don't understand how to factor it.

try some grouping:

c^2(a-b) - a^2(a+b) + b^2(a+1)

That pesky 1 at the end makes me think there's a typo. All the terms are degree 3 except that lone b^2. I suspect there may also be a sign mixed up somewhere as well.

Are you sure that's the way it is?