Prejudice is significant because it was what immigrants had to face due to the native-born Americans' fears?

Yes.

We could also argue that prejudice hurt the prejudiced people as well as their victims.