I don't understand your problem. It's over 5% so use the quadratic formula. I obtained 0.0113 which isn't that far from 0.01166 but 0.01133 satisfies the equation better than 0.01166. Technically, we're splitting hairs because you can use only 2 significant figures anyway since Ka isn't known any better than that. That being said, there isn't that much difference between 0.012 and 0.011.
Are we allowed to set x = 0 if it is like 5.83%?
Which x are you talking about?
The equation, I assume is
x^2/(0.2 - x)] = 6.8 x 10^-4.
You may NOT set the x of x^2 to zero else you can't solve the equation. Setting the x in the denominator to zero is allowed if the 5% rule (or whatever your prof sets), is not exceeded. In this case, it is exceeded so it is not allowed. [By the way, a better way of saying it is "assume 0.2-x = 0.2" and that way it isn't confusing which x we are talking about.] And for whatever it's worth, I always told my students to use 10%; however, Ks can be determined with better accuracy now so the 5% rule is a little more strict. Technically, it doesn't matter much except where x is large compared to the molarity or where Ka is large such as 10^-1 or 10^-2. Another thing to look at is that often teachers want their students to get practice at (a) solving the quadratic and (b)determining when the quadratic is appropriate so setting a smaller percent gets those two purposes accomplished. You can see how much difference those two numbers make if you substitute 0.01133 for x (and use 0.2-x) and see if the left side = 6.8 x 10^-4, then use 0.01166 and do the same thing. When I did this I came up with about 7.22 x 10^-4 when I used 0.01166 and it was 6.80 x 10^-4 when I used 0.1133. That's a 5.8% error when using 0.01166 (but you already knew that). I hope this helps.