Determine the (binomial) coefficient.

(x^2+3)^12; ax^8

I got 12C8 * (x^2)^4 * (3)^4 which is
40,095*x^16 but my teacher said it was incorrect.

A similar problem to it was
(x+3)^12; ax^5 where I determined the answer to be 1,732,105*x^5 which was correct.

your answer contains the factor x^16, but it was supposed to be x^8

so you would need (x^2)^4 to be in your expression.

term(5) = C(12,4) (x^2)^4 (3)^8
= 495 x^8 (6561)
= 3247695 x^8

verification:
look for the term containing x^8 in

http://www.wolframalpha.com/input/?i=expand+(x%5E2%2B3)%5E12

formal way:
have you learned that the general term of the expansion (a+b)^n is
term(r+1) = C(n,r) (a)^(n-r) (b)^r ?

so for yours:
term(r+1) = C(12,r) (x^2)^(12-r) (3)^r
= 3^r C(12,r) x^(24-2r)

but you want 24-2r=8
-2r=-16
r=8

and our coefficient is 3^8 C(12,8)
= 6561*495
= 3247695 ---> your value of a