If the drift velocity in a conductor is so small, then why does an auto battery influence the starter as soon as you turn the ignition switch?

The drift velocity is only small when no voltage difference is applied across the conducting wires; the battery's potential difference greatly increases the drift velocity.
There is no gap between the battery terminals and the starter, so all electrons moving under the influence of the battery's potential difference are inside the starter.
The battery's potential difference generates an electric field across the starter, and electrons throughout the starter begin to flow, creating a current throughout.

Even though the drift velocity of individual electrons is small, the sheer number of electrons in the conducting material allows for a significant flow of charge when a voltage difference is applied. This flow of charge, or current, is what allows the starter to be activated almost immediately after the ignition switch is turned.