I see the problem now. I was confused, I didn't read your post carefully enough. I thought you had two sets of speakers, but you are switching the same speaker from one channel to another, so now I understand your concern, which is valid.
In that case it would be better to have a center OFF switch, or two separate switches such as the knife switch you posted. Two switches are probably better since it is less likely that you would accidentally cross connect the speakers. To be extra safe you should do the switching with the amp turned off, so even if you accidentally cross-connect it won't do any damage. If you always turned the amp off during switching then you could use a ON-ON switch with no center OFF.
To be absolutely safe you would need to do an interlock type switch but that would require a more complex circuit, such as relays controlled by a single switch.