ideally there is a balance. For example at some point usually around 30mm Xmax one way you start running out of benefit for a reasonable cost by trying trying to increase xmax anymore. Also you need to consider that at certain point xmax doesn't matter sense higher frequencies upper bass for example is about surface area of a driver ,number of drivers, power handling ,and power the amplifier can deliver.
Your given example doesn't make sense getting a driver to 50mm xmax one way is waste of time. It can be done but it will very expensive and you might consider more drivers or a big driver.
In general driver size isn't a choice. Driver size is usually dictated by box size, power available, budget and generally the type of subwoofer you want to make (sealed , ported ,horn, ecte) and again size.
For example generally the biggest quality driver makes sense. But for example many don't use 18's because their really big and require a lot of box size. So for example people might use two 15 inch drivers to get in the same ballpark of surface area.