Posted by jack on Tuesday, June 18, 2013 at 2:42pm.
I assume you mean
cot^2 A(1 - 3 sec A + 2 sec^2 A)=1
If so, then
1 - 3 sec A + 2 sec^2 A = tan^2 A
sec^2 A - 3sec A + 2
(secA-1)(secA-2) = 0
secA=1 so A=0
secA=2 so A=π/3 or 5π/3
Looks like 5π/3 is the only solution > π/2
Check:
cot^2 5π/3 = 1/3
1 - 3 sec 5π/3 + 2 sec^2 5π/3 = 1-6+8 = 3