In the past, what makes someone or a group of people considered to be "unfit" members of society?

Please see my previous answer to your question.