Two protons, an infinite distance apart are brought together to a separation of 1 nm.

What is the initial potential energy of the system?

Usually you define the potential energy as zero at infinity, then it goes up as kQ1Q2/r as they are forced toward each other

similarly if one were + and the other -, the potential would again be chosen zero at infinity and would be kQ1Q2/r again but since one of those Q values is negative it would be a negative energy.

Remember potential is relative and only difference matters. You can choose zero wherever you wish.