what does it suggest when someone is backing away besides frightened?

S/he doesn't want to be involved?

S/he is disgusted with the sight?
S/he disagrees with what was said?

for example in this drama. the phantom tollbooth. HUMBUG. backing away. "if it's all the same to you, i'd rather not

Well, fright is a good reason to "back away" from something. It might be backing away from an argument you know you can't win. It might be backing away from a person you have learned to distrust for some reason. There can be any number of reasons to back away from something or someone.

It sounds as though Humbug doesn't want to be involved.

In the play (which I have not seen or read), why do you suppose "Humbug" would back away or "rather not?" What is it that he would rather not do? There are, undoubtedly, clues in the script.

DISCHORD offers HUMBUG medicine that will allow him to never have to hear another beautiful sound again

I suspect Humbug just doesn't want to participate in such a thing!