How do you find the exact values of sin(u/2), cos(u/2), and tan(u/2) of tan(u)=-5/8

since the tangent is negative, angle u is either in quadrant II or IV.

I made a diagram of a right-angled triangle with a hypotenuse of √89

we know
cos 2A = 2cos^2 A -1 or 1 - 2sin^2 A

so cos u = 2cos^2 (u/2) - 1
-8/√89 + 1 = 2cos^2 (u/2)
(√89 - 8)/(2√89) = cos^2 (u/2)
cos (u/2) = ± √(√89 - 8)/(2√89)

for the sine:
cos u = 1 - 2sin^2 (u/2)
2sin^2 (u/2) = 1 + 8/√89
sin^2 (u/2) = (√89+8)/(2√89)
sin (u/2) = ±√(√89+8)/(2√89)

Now sorting out the ±

IF u was in quadrant II, then u/2 is in quadrant I and both the sine and cosine would be positive.

If u was in quadrant IV, the u/2 is in quadrant II
and the sine would be positive and the cosine negative.

Since tanx = sinx/cosx
for tan (u/2) divide the above results, using the appropriate ± values.
Notice since both the sin (u/2) and cos (u/2) have 2√89 in the denominator, that part will cancel when you do the division.