The patient was advised to return to the clinic immediately for any redness or swelling.

immediately modifies which word:

What is the patient supposed to do IMMEDIATELY?

The word "immediately" modifies the verb "return".