Why were immigrants generally distrusted by the American middle class during the late 19th Century?

Many of them looked upon anyone who was different (language, religion, customs) as undesirable. Some were also afraid foreigners would take their jobs at lower wages.