1. He is a my friend. (X)

2. He is a friend of mine.(O)
(Does 'mine' refers to 'my friends'?)

3. Look at my that book. (X)
4. Look at that book of mine. (o)
(Does 'mine' refer to 'my books'?

5. They are some my books.
6. They are some books of mine.
7. They are some of my books.
(Which one is correct?)

8. Take each my book.
9. Take each book of mine.
(Which one is correct? Do you have some other correct expressions?)

10. They are all my books.
11. They are all of my books.
12. They are all books of mine.
(Which one is correct? Do you have some other correct expressions?)

2, 4, 6, 7, 9, 10, 11, and 12 are correct.

Here are corrections for the others:
1. He is my friend.
3. Look at my book.
5. They are my books.
8. Take each of my books.

http://grammar.ccc.commnet.edu/grammar/cases.htm
from
http://grammar.ccc.commnet.edu/grammar/pronouns1.htm#personal