Originally Posted by
Bob Pariseau 
You have to set the Sub volume in the first step. You made no mention of that.
I presume you are using Stereo Signal FRONT LEFT/RIGHT and using the XLR outputs in lieu of the normal LF/RF RCA outputs from the multi-channel set. Meanwhile the Sub is wired from the RCA output of the multi-channel set.
For maximum accuracy, I suggest you use the LPCM test tracks from AIX Audio Calibration, Blu-ray (with an SPL meter) instead of the internal test tones, but the procedure is similar.
Suppose you decide to use the 5.1 LPCM test track. First set Down-mix to 5.1. Then set LF/C/RF/RS/LS to Large and Sub to ON (also LR/RR to OFF, although it won't affect what follows). You need to set Center and the 2 Surrounds to Large to avoid down-mix attenuation when playing the 5.1 test track (down-mix is the lesser of your Down-mix setting and the speakers you actually have turned on). Set LF/RF to -10dB volume trim (to adjust for the input sensitivity of your power amp). Leave Sub at 0dB volume trim. The volume trim for the other speaker outputs does not matter since they are not wired. Using the 5.1 LPCM test track, verify that LF and RF are producing the same output level. If not, then raise the volume trim a bit on the weaker speaker. Now adjust the volume knob on the Sub to produce the same output level you just established for LF/RF. The volume knob setting on the Sub will thus have adjusted for BOTH the needed +10dB boost (all speakers Large) and the 6dB difference between the XLR outputs and the RCA outputs. Note that volume setting for future reference.
Now, without changing anything in the OPPO, raise the volume knob on the Sub until its output is an ADDITIONAL +5dB HIGHER than LF/RF.
Now switch LF/RF to Small, and set C/LS/RS to OFF. Set Stereo Down-mix. Set a Crossover frequency. (The correct choice of Crossover frequency will depend upon the capabilities of your speakers and Sub and how they couple to the room. Start with 80Hz if you are still figuring this out.)
You are done. The additional +5dB boost you added to the Sub in that last adjustment takes care of the +15dB boost the Sub needs with LF/RF set to Small. (The difference between the XLR and RCA outputs is also still accounted for in that setting.)
Now here's an additional refinement: Unless you are in a large listening room, or a room with extensive bass treatments, a "desirable" room response characteristic called Room Gain comes into play -- one of the aspects of perceiving sound as being played in a good listening room instead of, say, outdoors. Room Gain has the effect of boosting bass frequencies a few dB -- 2 to 4 dB is typical. And so the most pleasing adjustment of the Subwoofer volume may actually be a few dB HIGHER than the precise match to LF/RF described above.
To experiment with that, set the Sub about +2dB higher than LF/RF in the first step -- +7dB higher in the 2nd step. Do some listening and see what you think.
Bass response is *HIGHLY* dependent on how the Subwoofer (and the woofers of LF/RF) couple to the room. There are tools you can get that let you chart the bass response of your system at various mic locations around the seating. Room problems can result in resonance peaks at certain frequencies and cancellation nulls at other, nearby frequencies. Thus proper bass setup can be a complicated task. But the above volume setting stuff is the starting point. Do re-check this stuff if you move the speakers or Sub -- even inches matter for bass frequencies.
Finally, if you want to try SACD DSD-Direct-to-Analog Conversion, you should ONLY play the Stereo layer of your SACDs. There is no audio processing possible and so if you play the 5.1 layer the extra channels just get discarded. Now when doing this -- again because there is no audio processing -- the speakers are treated as Large (no Crossover). And so there will be NO SUBWOOFER OUTPUT. There's no LFE channel in the 2.0 layer of course, and there's no bass being steered to the Sub from the mains. The upshot is that you don't have to fiddle with the volume setting on the Sub because the Sub will be silent! Voila!
--Bob