Why not just us a standard A/B speaker selector switch, but wire it the other way around?
Put the speakers on the "input" and one amp on Speaker "A" and the other on Speaker "B".
Put the speakers on the "input" and one amp on Speaker "A" and the other on Speaker "B".