What is the [H+] of a 0.1 M malonic acid with a Ka of 10^-9 solution?

I thought it was just (Ka x Ca)^(0.5), but that doesn't get me a pH of 5.

You've tried to take a shortcut and made a simple mistake. Take your time and set these up from the beginning.

...........HM ==> H^+ + M^-
initial.....0.1....0.....0
change.......-x....x.....x
equil......0.1-x...x......x

Ka = (H^+)(M^-)/(HM)
Substitute from the ICE chart I've prepared and solve for x, then convert to pH. The answer is pH = 5.