More expensive drivers is usually a combination of effects that drive up costs, not because they necessarily sound 'better.' Better build quality, more power handling, better cooling, better inductance control, better linearity ... and the list goes on.
The PA460 is super cheap, if you look at how much material goes into it you can see why, its just a basic driver with quite a bit of xmax. I think it was MTG90 or maybe somebody else that look at the distortion at high xmax for the driver and it showed to be very clean which is why the VBSS is so popular. If it dug low and put out tons of distortion, price be damned, it probably wouldn't of become so popular.
As for starting out with more expensive drivers, well that totally depends. The UM18 is a middle of the road price 18" driver, there are obviously better drivers out there than the UM18 but not in the price range. Those drivers being better doesn't mean they will be different or way different at the same sound levels in the same box, just that they can do MORE than the other.
The PA460 in a devestator does well, it already gets louder than what anyone person would need in a single living room, but you might have to play with the DSP a little bit when compared to the Lavoce 21" sub.
I guess long and short answer is, it depends. You won't know if those better drivers actually sound better until you sim to make sure they work in the enclosure and then listen to them side by side.