What should be the number of significant figures in the following calculation?
It's an error calculation which is
absolute value (True value - experimental value) / true value * 100
so it would be:
absolute value [-601.7 - (-587)] / -601.7 * 100
Should the answer be 2.44 or 2.4?
There's no decimal point when subtracting the two values, so it would be a 3 sig fig number, divided by a four sig fig number, so it should just be 3 sig figs, regardless of decimal point due to the final division right?
I don't think so.
-601.7
+587.
----------
..14.7 and since the 7 doesn't count, there are only two significant figures.
Then 14.7/601.7 = 0.02443 or 2.4% error. There are those who will round 14.7 (the original subtraction) to 15, then
15/601.7 = 0.02493 which rounds to 2.5%.
I expect 2.4% is the number you want.