This is a limiting reagent problem. I know that because an amount for BOTH reactants is given.
2NO + O2 ==> 2NO2
mols NO = 0.866
mols O2 = 0.503
Convert mols NO to mols NO2. Use the coefficients. That's 0.866 x (2 mols NO2/2 mols NO) = 0.866 x 2/2 = 0.866
Convert mols O2 to mols NO2. That's
0.503 x (2 mols NO2/1 mol O2) = 0.503 x 2/1 = 1.006
You know both answers (0.866 and 1.006) can't be right and one of them must be wrong. The correct answer in limiting reagent problems is ALWAYS the smaller value and the reagent producing that value is the limiting reagent. Therefore the NO is the limiting reagent and you will produce 0.866 mol NO2.