1. The new automatic teller machine did not function as expected.

2. The new auto teller machine did not function as expected.

3. The new automated teller machine did not function as expected.

(Which expression is grammatical? ATM)

#3 is correct.

"Automated" is the correct term.
http://en.wikipedia.org/wiki/Automated_teller_machine