how would an increase in mass inside of a rollercoaster affect the sturdiness of the coaster
Hmmmm. I wonder what you mean by "sturdiness"?
If you mean stability, then it is a problem. The mass inside the coaster is higher than the wheels, and centripetalforce acts on the mass...one would like it to work on the wheels at the track level. If it is higher, than it forms a torque on the car which can flip it.