If I was to say, "Please write an "M" on your paper." Why isn't it a "M" since "M" is not a vowel?

If you were to write the word 'M', you would spell it em. The first sound is a vowel sound. So you would say "an M"

Also, when using quotes within quotes, use single quotes. "Please write an 'M' on your paper."