1. I don't have any books.
2. I do not have any books.
3. I have not any books.
[Which one is incorrect? In British English, do you use #3 frequently?]
1 and 2 are fine. I don’t think anyone would use #3. Possibly this, though: “I have no books.”
Out of the three sentences you provided, the third one, "I have not any books," is grammatically incorrect in standard English.
In British English, the more common and preferable way to express the same idea would be "I do not have any books," which is the second sentence you provided. The use of "do not" to form a negative sentence is widely used in both American and British English.
In general, the construction "I have not any" is less commonly used in modern English. Instead, it is more natural to use "do not have" or "haven't" to negate the possession of something. So, to answer your question, the incorrect sentence is number 3, and in British English, sentence number 2 would be the preferred way to express the idea.