Hmm, I have installed ac3filter on both of my two Vista computers with no problem. Not sure why that is failing for you.
In general it's a good idea to shut down the User Accountr Control in Vista while troubleshooting and installing stuff. When everyting is the way you want it, turn it back on.
You say that you try to get sound trough your HDMI connection, however on your first image the Speaker device is set as the default device. This means that the only sound you will get out is trought the analog outputs, problably from you motherboard.
Kindof stupid question, but have to tried to set "Digital Outputenhet (HDMI)" as the default device? The default device is the device that all sound normally will be passed on to as long as the software you are using does not have special options for this (something it normally does not have). This also applies to Vista Media Player and Vista Media Center.
If that does not do it for you, I would have tried to connect the PC trough the SPDIF connection (light or coax) to your receiver and set the "Reltek Digital Output" as the default device. The problem might be in HDMI 1.3 issues (the standard is relatively new).
I have not looked into the HDMI options, but for the Digital Output you have to go to the "Egenskaper" and the tab "Understøttede formater" and select the formats that you receiver supports (Normally DD/DTS and sometimes also WMA).
From here you may also play test-tone to check if it actually works. If you do not get any sounds from the test tones in this screens there is no point trying anywhere else.
I also assume that you have all updates from Windows Update installed?
You may as well try to just uninstall all reltek stuff on your PC and just use the native Vista Drivers.
You should also try to go www.giga-byte.com
and download all the latest drivers and chipsets for your motherboard and install them. This will also include drivers for audio.
And of course, go to www.ati.com
and download the latest drivers to your graphics card. They relesed a new version (8.1) just a couple of days ago.
Hope you get it working...
Just so you know this: If you receiver supports DolbyDigital and DTS and you are using digital out (HDMI or Coax) there is no need for AC3 or any other software decoder. This should not do any difference as the signal should be decoded by the receiver, not the PC.
Focus your testing and troubleshooting using the sound settings (from the first screenshots shown). That should allow you to test anything needed.
If you are using analog output then it's a different case because then either the soundcard (motherboard in your case) or software must to the decoding.