Putting phantom-imaged sounds into the front soundstage is one thing but placing them into 5.1 is quite another.
Just as sending phantom imaged centre info to the centre speaker is not arbitrary, neither is the information sent to the surround speakers.
One of the early Stereophile magazine test CDs has a phase test that switches between correlated (in-phase) and decorrelated (out of phase) bass guitar. The in-phase signal images dead centre of the soundstage. The out of phase track doesn't image in the front soundstage, appearing more like it's indistinctly around you.
This decorrelated info is what is sent to the surround channels; nothing else. It's not like some random guitar sounds that would normally image up front are re-routed to speakers behind you for no apparent reason.
Musical intent is not the issue, imho, since we are dealing with what the mixing and mastering stuffed into 2 channels without a clue as to how.
Sure we have a clue. In fact, the recording is loaded with clues, which the surround processing relies upon for steering instructions.
A trumpet that is panned left of centre will still image there whether the soundstage is made up of 2 or 3 or 5 speakers. Sounds that normally image outside the front L/R speakers will be steered more towards the surrounds to maintain and stabilize that imaging.
It's the same problem I have with the technicall impressive Penteo process: the result may (or may not) be pleasing but it has no basis in reality.
I agree as far as moving front soundstage content to the surrounds. But the 2-channel to 3-channel Penteo process is different and maintains front soundstage imaging to a fault. It has no choice since it is the 2-channel mix that dictates what will end up in the left, centre and right speakers.
Finally, please understand that I'm not trying to convince you to use surround processing. I consider personal preferences to be exactly that: personal. However, I do hope you understand that what surround processing is attempting to do is not arbitrary. The only thing sent to the centre speaker is information that would have imaged there anyway. The only thing sent to the surrounds is stuff that would have imaged outside the front soundstage to begin with. Those clues are in the 2-channel recording, so there's no need to steer sounds randomly when it can be consistent with the 2-channel mix. To that end, it is analogous to how a bitmap image or video scaling works.