In the past, what makes someone or a group of people considered to be "unfit" members of society?

Wrong religion

Wrong family
Wrong color
Wrong nationality
Wrong job
Wrong income bracket
Wrong sexual orientation
Wrong "morals"
Wrong neighborhood
Wrong level of intelligence