Ahh -- one of my favorite stories.
Yes, the city is New York. The sweet food in jars could be several things. I've always thought of honey or jelly.
This story is set in the future, presumably long after a nuclear war has devastated everything. The narrator of the story is descended from survivors of this war who essentially began civilization all over again. They are highly religious and superstitious. They've been warned away from New York because of the radioactivity that appeared to have been present for generations after the war.
Everything is real, but the narrator's interpretations of what he sees is often symbolic. He didn't know whether the statue of Washington represented a man or a god, but he prayed anyway -- just to be on the safe side.
This site may give you a little more information.
Yes, New York City. Sweet food in the Jars? Does it remind you of Eve and the Tree of knowledge? Could it be the sweet food is knowledge?
Yes, everything is symbolic.
The theme is that man can destroy himself with knowledge, and that man's destiny is in his own hands.
As you can see, different people interpret this story (and other pieces of literature) in different ways. I'm sure Bobpursley is right that there is a lot of symbolism in "By the Waters of Babylon."
As Writeacher told me, "There are no 'right' answers" to some literature questions.
What do YOU think?