I can't find my English book. Why don't you check your school bag?

I tried that. Anyway, thank you.

(What does 'that' refer to? Does 'that' refer to 'checking my English book'?)

"That" refers to checking my school bag.

Yes, in this context, 'that' refers to 'checking your English book in your school bag'. So the person is saying that they have already checked their school bag for the English book and it was not there.