I just tried both RGB settings.
Over component (as always), I can see the BTB bar on the extreme left of the HD-DVD DVE pluge pattern.
On HDMI->DVI to my Sony, I can't, regardless of whether the RGB setting is Standard or Enhanced. I even tried rebooting the player in between setting changes.
Now, since firmware 2.1, the black levels have been better on HDMI->DVI for me (movies are at least watchable). So, when I watch HD-DVDs, I use component; when I'm upconverting SD-DVDs, I use HDMI->DVI.
But it doesn't seem (at least for me) that the RGB setting fixes anything for me.