calculate the molecular formula of a compound with vapor density of 30 having 40% carbon, 6.67% of hydrogen and the rest is oxygen.

40%C

6.67%H
53.33%O [100-(40+6.67)] = 53.33

Take 100 g sample which gives us
40 g C
6.67 g H
53.33 g O.

Convert to moles.
40/atomic mass C = ??
6.67/atomic mass H = ??
53.33/atomic mass O = ??

Now find the ratio of the elements to each other with the smallest number being 1.00. The easy way to do that is to divide the smallest number by itself which MAKES it 1.000. Then divide the other numbers by the same small number. This will give you the empirical formula which you should find to be CH2O.
The molecular formula is found as follows:
Find the empirical mass. That is C + 2*H + O = 12 + 2 + 16 = 30.
molar mass = 2*vapor density = 2*30 = 60
molar mass/empirical mass = n; therefore, the molecular formula is
(CH2O)n.