An art teacher is ordering colored pencils for the new school year. The teacher buys 24 small boxes, 12 medium boxes, and 5 large boxes. She has to return 3 small boxes due to defects. How many colored pencils does the teacher have in all?

The teacher bought 24 small boxes - 3 defective small boxes = <<24-3=21>>21 small boxes.

The small boxes contain 21 small boxes x 12 colored pencils = <<21*12=252>>252 colored pencils.
She also bought 12 medium boxes x 24 colored pencils = <<12*24=288>>288 colored pencils.
The large boxes contain 5 large boxes x 36 colored pencils = <<5*36=180>>180 colored pencils.
At the start of the term, she had 252 + 288 + 180 colored pencils = <<252+288+180=720>>720 colored pencils. Answer: \boxed{720}.