A golfer takes two putts to sink the ball, one is (81.6 ft, 31.7 degrees N of E) and the other is (3.20 ft, 53.4 degrees W of N). What is the displacement of the single putt that would sink the ball on the first try?

Well, what I did was make a diagram of the two vectors,
I named the first one Alpha and tried to find both of their components.
Alpha-subX (for the x component)
Alpha-subX = 81.6 ft * cos(31.7) = 69.426

Alpha-subY (for the y component)
Alpha-subY = 81.6 ft * sin(31.7) = 42.878

I also did the same thing for the 2nd putt, and named it Beta.

Beta-subX = 3.20 ft cos(53.4) = 1.907
Beta-subY = 3.20 ft sin(53.4) = 2.569

I "think" I'm supposed to combine the x components with each other

Alpha subX + Beta subX
and Alpha subY + Beta subY

Yes, exactly BUT THE SECOND ONE was west of north

x distance = 69.4 -3.2 sin 53.4 = x
y distance = 42.9 + 3.2 cos 53.4 = y

d = sqrt (x^2 + y^2)

tan of angle north of east = y/x

What do you mean, I did indicate on the question that it was west of north?

But going west is NEGATIVE x

Why did you swap Betas trig functions?

This is right:

Alpha-subX (for the x component)
Alpha-subX = 81.6 ft * cos(31.7) = 69.426

Alpha-subY (for the y component)
Alpha-subY = 81.6 ft * sin(31.7) = 42.878

This is WRONG

Beta-subX = 3.20 ft cos(53.4) = 1.907
Beta-subY = 3.20 ft sin(53.4) = 2.569

I CLAIM
Beta-subX = -3.20 ft SIN(53.4) = 1.907
Beta-subY = 3.20 ft COS(53.4) = 2.569

Draw a picture

Because the angle 53.4 is left from the Y axis, not up from the X axis.

They tricked you with the west of north
that is left of the Y axis
DRAW the picture.

How do I know which trig function to use?

I made my second triangle like this
_____
\ |
\ |
\ |
\|

Wait that's not how it's supposed to look.

Ok

copied your result, should be

I CLAIM
Beta-subX = -3.20 ft SIN(53.4) = -2.56
Beta-subY = 3.20 ft COS(53.4) = 1.90

when the angle is in quadrant II

53.4 degrees left of north
then east component is - sin
then north component is + cos

if you want the angle from the x axis then it is
90 - 53.4 = 36.6
then you could use cos and sin like you want because
sin (90-x) = cos x
cos (90-x) = sin x :)