In the binomial series from (a+b)^5, the powers of a decrease by 1 each term, and the powers of b increase by 1. If you carry on this pattern beyond te "end" of the series you get

a^5 + _a^4 b + _a^3 b^2 + _a^2 b^3 + _a^1 b^4 + _a^0 b^5 + _a^-1 b^6 + _a^-2 b^7 +...

where the spaces are for the coefficients.

a. What will be the coefficients for terms beyond 8? How do you know?

First of all there would only be 6 terms in the expansion using the exponent of 5

Are you familiar with Pascal's Triangle?
If not, google it, and you will find that you need the coefficients of
1 5 10 10 5 1
that is:
a^5 + 5a^4 b + 10a^3 b^2 + 10a^2 b^3 + 5a b^4 + b^5

these coefficients can also be written in the form
C(5,0) C(5,1) C(5,2) C(5,3) C(5,4) and C(5,5)

where C(n,r) is defined as n!/(r!(n-r)!)
since n! is only defined for whole numbers of n
expressions such as C(5,6) are undefined and don't exist, so only 6 terms would exist for your expansion.