Why is the second not defined by the length of day? Is it because it's such a small fragment of time?

The length of a day? How would that be determined? think on that. Remember the length of a day varies in orbit. We could use a "mean" solar day, but that is as useless as determining "mean" tide.

Time is something that we need a reference to that anyone can get the precise standard, so we use atomic standards, which do not vary.