1. I can't believe that he said.

2. I can't believe that he said yes.
3. I can't believe what he said.
4. I can't believe what he said yes.
------------------------------------
#2 and #3 is right. The others are ungrammatical? Is that right?

Yes, you're right -- 1 and 4 are not correct.