Here is what I had to do to get my PC to play back from my Yamaha correctly;
On the PC, set the speaker configuration to "Stereo" and set the playback device to your HDMI video card (or whatever is outputting your HDMI signal)
The reason for this is the signal is being outputted "PCM" and windows doesn't do internal decoding. Now your video player (VNC, Windows Media Player, Power DVD) may do decoding but it will still send it out as a PCM signal. At that time the Yamaha will sense the signal and do one of two things, 1. switch to the proper sound format i.e. Dolby Digital, Dolby HD ect. and give you surround sound or just ouput stereo and you will have to tell the Yamaha to decode the signal i.e. DSP modes. This has been an issue with Windows for years and as an IT tech this was the only way I found to resolve the issue. One thing I do is insure my PC is set to play back 24Bit 4800hz as a minimum. You can select a higher bitrate but you will want to test higher playback resolutions to make sure your receiver can decode them properly.
This is a known issue since windows XP and you are not alone in dealing with this issue but the above does work. FYI if you are running windows media center, under speaker settings you will also want to make sure your speakers are set to stereo and the input is HDMI for video. If you have the speakers set to 5.1 or 7.1 you will get test tones, but when you playback audio you may still only get back stereo. bottom line tell the Yammy what DSP you want to use and then it will work correctly.
Hope this helps eliminate your frustration.
Yamaha CX-A5000, ATI AT 2007 Amp (7X200W), ATI AT 2004 Amp (4X200W)
Klipsch RF 63's - Main, RB 61's - Front heights, RC-62 - Center, RS-62's - Surround, RS-52's - Rear Surround
2 Rythmik DS-1510 DIY Subwoofers (With PEQ600XLR3 Amp) each connected with Antimode 8033S-II