The equation for density is mass divided by volume. An increase in density can result from all of the following except:

A.an increase in volume with an increase in mass
B.a decrease in volume with no change in mass
C.an increase in mass with no change in volume
D.a decrease in mass with an increase in volume

B and C will increase the density so B and C can't be correct.

D Making the mass smaller and volume larger will DECREASE the density; therefore I would pick this as the answer.
What about A. That is confusing and in my opinion should not be listed as a possible answer. Why? Because if you increase mass and increase volume, those can be varied to make density increase, or decrease, or stay the same; i.e., A COULD be the answer if the variation was right. For example, let's say mass is 3 g and volume is 3 cc so density = 3/3 = 1 g/cc. Now let's increase mass to 4 g and increase volume to 5 cc so the density is 4/5 = 0.8 g/cc which is smaller than the 1 g/cc which then could be a correct answer to the problem. Personally, I think whoever made up the problem failed to see that but they intended D to be the correct answer. I would definitely not confront your teacher with this. Just answer D and go with the flow.