the first three erms in the expansion of 1+ay)^n are 1, 12y, and 68y^2.

(1+ay)^n= 1+nay+n(n-1) (ay)^2
2
evalate a and n. using that fact

so nay = 12y, which gives you na = 12, and

n/2(n-1)a^2y^2 = 68y^2
which gives us n/2(n-1)a^2 = 68
n(n-1)a^2 = 136 square na=12 to get n^2 a^2 = 144 and sub

n(n-1)(144/n^2) = 136
reducing this and solving I got n = 18
then a = 12/18 = 2/3