what's the value of A if cot^A(1 - 3 sec A + 2 sec^2 A)=1 and A>90 degrees.

I assume you mean

cot^2 A(1 - 3 sec A + 2 sec^2 A)=1

If so, then

1 - 3 sec A + 2 sec^2 A = tan^2 A
sec^2 A - 3sec A + 2
(secA-1)(secA-2) = 0
secA=1 so A=0
secA=2 so A=π/3 or 5π/3

Looks like 5π/3 is the only solution > π/2

Check:

cot^2 5π/3 = 1/3
1 - 3 sec 5π/3 + 2 sec^2 5π/3 = 1-6+8 = 3