I'll just say "there's more to it than size" -- so much more.
Reasons to chose a difference size woofer driver (that aren't just about cost):
- Drivers of different size "beam" at different frequencies, it's important to match the driver with its tweeter via the crossover. Many "cheap" speakers may just change the driver (to differentiate the models) without caring at all about how it changes the tweeter/mid crossover integration.
- Smaller drivers can't move as much air without more excursion, more excursion usually means more distortion (this matters a bit less at sub frequencies, and a good motor can overcome some of it).
- Due to the above, as has been mentioned, smaller drivers tend to play with less "authority" than larger drivers (but, if you stick a 10" mid-range into a speaker without a proper wave-guide on your tweeter, there's a really good chance you'll have some nasty off-axis response due to point #1
And that's probably just scratching the surface and off the top of my head.
So, here's what I'll say: listen! with your ears, in your room, at your desired volume levels. If you like something with a 4.5" driver and it satisfies your needs for volume and sounds good to you, great! If that same manufacturer offers a similar speaker with a 5.25" or 6.5" driver, you'll need to figure out from measurements or listening tests to see if they accounted for any/all of my above points -- I'd say it is a coin-flip whether they cared to or not. However, if they change nothing else, there's probably one "ideal" driver size that integrates best with the tweeter and the rest are compromised to leverage economies of scale (e.g., reuse the xover across the model line despite it being compromised with a different driver -- maybe tweak a component to make it a lesser compromise, but probably not redesign it for "perfection").