You won't compromise sound quality, but you need to be aware of the load you are presenting to the amp - most amps like to see between 4 to 8 ohms per channel. There are lots of options for volume controllers that handle two pairs of speakers - many of these have jumpers to allow the appropriate loading to be dialled in, but given your post, it sounds like you've already got one.
Depending on how you were intending to wire the speakers together, you should be able to present the amp with an acceptable loading. For example, wiring a pair of 8 ohm speakers in parallel gives a total restistance of 4 ohms - but you wouldn't want to put the two in series as this would create a 16 ohm loading, which is getting outside of what an amp typically likes to see.
Check the specs of your amp, switch and speakers to determine where you need to get to - I can certainly try to help if you want to post more info.