I can never quite get my who and whom right. Does this sentence make sense?

This brought him farther from those whom he loved and those who loved him.

Yes. You're right.

Whom in your sentence is the object of loved.