I've done a lot of experimenting, trying to nail down this issue. And I've come to the conclusion that the 5300ES doesn't output the correct subwoofer level when using PCM. It's about 4dB too weak.
I experimented using the Avia SD DVD, as its subwoofer calibration tones are bass sent to the individual 5 channels that depend on bass management in the receiver to re-route bass.
I used the PS3 for most of this, since it's easy to change between PCM (internal decoding) and bitstream (sending DD to the receiver as-is) on the fly, but I reproduced this with my Denon 2910 upconverting player, swapping between PCM and bitstream.
Test 1 -- PCM versus Bitstream, standard bass management (xo around 80Hz, receiver re-routing bass to sub)
*Bitstream = 80dB
*PCM = 76dB
Test 2 -- PCM versus Bitstream, standard bass management, speaker wire unplugged so only sub plays
*Bitstream = XdB
*PCM = 4dB lower
(don't remember the exact number)
Test 3 -- PCM versus Bitstream, speakers set to Large with no bass management engaged, only speaker playing bass
*Bitstream = 80dB
*PCM = 80dB
So I'm guessing the receiver outputs the sub level 4dB weaker when using PCM. It could also be a bass re-routing issue, but I didn't test any pure .1 LFE signals to check PCM vs bitstream. My HD DVD version of DVE might have a pure LFE signal, but I don't feel like testing right now.
I've just set up a "PCM" profile/position that has the sub level set 4dB higher, for mainly the PS3, since I can bitstream HD DVD off of the A35, and SD DVD off of the Denon 2910. Don't want weak bass on those Blu-ray movies...
Has anyone else encountered this? Is it a known issue with the receiver? Am I just missing some deep, hidden setting in the receiver that fixes this? I've found a simple workaround, but it's still a pain in the butt to have to change just for PS3.