I think you are right where I was with my Denon 3805, except the Denon does not have HDMI. If you have HDMI 2 connected to your AV8801 and you go to Setup to Audio format and select HDMI audio "auto" for 7.1 movies. If you want to use your Rotel for a stereo SACD then connect the stereo RCA outs to your Rotel. In Setup go to Audio Processing and keep dropping down until you get to "stereo signal." Select "stereo." Now, I think you will have to go to "Speaker Configuration" and downmix to stereo when you want to listen to SACD or any stereo audio source just through the Rotel and power down the AV8801. If you watch a movie select 7.1 downmix and power down the Rotel. This should give you some things to try. For myself, I found that the Oppo DAC's were so much better than the Denon 3805 that I took the Denon out of the loop completely. I'm directly connecting the 105 analog outs to my Rotel for the side speakers, a bridged NAD for the center speaker and I'll run the stereo outs to a CJ MV45 tube amp for the Front R/L speakers (to do this you change the stereo out from "stereo" to Front L/R) plus a powered sub. I have a downmix to 5.1.
Meffy wants to be able to use ALL of the outputs into the AVR. He can't do it without unplugging wires and reconfiguring.