Can the 5% approximation be used to determine the pH of the following solutions? How do I do this question?

(A) 0.56 M hypochlorous acid - yes

(B) 3.00 M nitrous acid - yes

(C) 0.010 formic acid - no

Whether the 5% rule even applies or not; however, depends upon the concn of the acid and the value of Ka.

The Ka expression for an acid is
Ka = (H^+)(A^-)/(HA) and when we substitute an ICE chart into that we get this
Ka = (x)(x)/(C-x)
The 5% rule says that we can neglect x in the denominator (not in the numerator) IF H^+ is < 5% of C. So you must slip in some numbers and try it.
For HClO we have
Ka = (H^+)(ClO^-)/(HClO)
So we solve with the ICE chart
(x)(x)/(0.56-x) = 3E-8 (but confirm this constant in your text or notes).
A quickie calculation yield x = 0.00013 which certainly is negligible in caparison with 0.56 (5%C is about 0.03 and that is less than H^+.

HNO2 ==> H^+ + NO2^-
(H^+)(NO2^-)/(HNO2) = Ka
5.1E-4 = (x)(x)/(3-x)
A quickie calculation gives x = 0.039
5% of 3 is 0.15 and 0.039 is less than that so we can get away without solving the quadratic. Just another way of saying 0.039 is small and we can ignore when compared to 3 M.
Formic acid.
HCOOH ==> H^+ + HCOO^-
Ka = (H^+)(HCOO^-)/(HCOOH)
1.77E-4 = (x)(x)/(0.01-x)
A quickie calculation gives 0.0013
5% of 0.01 is 0.0005 so this can't be ignored since H^+ is >5%C (It's >0.0005)
You can see that 0.01-x = 0.01 if we ignore x but it's 0.01-0.0013 = 0.0087 if we don't ignore it.