how do each protagonist from one flew over the cuckoos nest,stranger in a strange land and catcher in the rye handle costom rules,behaviors of their respective societies
If you have read these books, you should be able to respond to this question. It sounds like an essay question, possibly for the end of the semester, so you need to make it good!
The protagonist is the main character in each story.
Customs, rules, and behaviors in a society are those things that people do that set them apart as a culture. Customs and behaviors are normal and traditional actions of people; rules, of course, are the laws and regulations set up by that society.
So ask yourself ... did each protagonist act within the particular society's boundaries? Or did he/she act out against them and/or fight them?
Once you've decided those things, you can start brainstorming -- mostly you'll need examples from each story that back up your claim about each of the characters.
http://www.sparknotes.com/lit/cuckoo/
http://www.sparknotes.com/lit/strangeland/
http://www.sparknotes.com/lit/catcher/