if a=√5+1/√5-1 and b=√5-1/√5+1 , find a^2+ab+b^2/a^2-ab+b^2

did you mean

a = (√5 + 1)/(√5 - 1) and b = (√5-1)/(√5 + 1) ??
or the way you typed it