I don't think you need to worry about sound quality. A switch would have almost no impact on a speaker connection, unless you are pushing so much power that the switch can't handle it and burns-up.
You might have to worry about impedance. That switch will put the speakers in parallel if both are enabled at the same time. If both sets of speakers are 4 ohms, the resulting impedance will be 2 ohms, which some amps won't be able to handle. A few amps can't handle 4 ohms, so even two sets of 8 ohms speakers could over-tax the amp.
If you never enable both sets of speakers at the same time, then you never have to worry about it.