As far as I know, the 3806 is HDMI 1.1 and lacks lossless decoders. The optimal setting for the player would be PCM Multi-Ch.
However, when set to DTS Re-encode, the player should not be sending stereo unless the source material is actually stereo.
Sometimes the audio handshake with the TV forces the source device to downmix and send stereo instead of multichannel. I believe the Denon has a setting to disable the HDMI audio output, which would solve that problem if, in fact, that's what's causing the stereo output. Or, as a test, you can unplug the HDMI connection to the TV and see whether you start getting 5.1 audio.