Do you have a decent switching box ? I was trying to find one with remote control and banana plug connections for less than a bazillion dollars with no success, so I was thinking about building one.
Might use a flashlight for remote control -- 3 flashes selects speaker #3 etc, or I guess I could make it the first "thing" (from Internet of Things) to go in the house since the local "essential services" policy prevents me from networking the coffee maker.