Anyway, the video files I am using to test are encoded with different audio tracks. DTS HD and Dolby digital 7.1 etc..
There is no Dolby Digital 7.1. The max is Dolby Digital EX, which was 6.1 and included a discrete surround back channel. Most receivers/pre-amps that do "7.1" with a Dolby Digital signal, perform extra processing to derive the 7th channel.
I am pretty sure SPDF coax can't do 7.1, 6.1, etc...
7.1 - No
6.1 - Yes.
That is why the HDMI through my video card did do it.
Not necessarily. It's a software configuration issue. With SPDIF, you have video going out through one mechanism and audio going out through another. Both need to be configured correctly. HDMI is much simpler and requires only a single configuration.
and I also think most games use dolby digital live to encode 5.1 audio which SPDF wouldn't be able to do either.
Why not? DD is DD, whether encoded live or encoded on a disk. And it does work.
I have a 5.1 speaker set up, however if I am viewing files or content that is above 5.1 say DTS HD 7.1 I will need to use HDMI to properly get 5.1 because it will downgrade it to a 5.1.
Wrong. Yes, you need to down convert to 5.1, but that has nothing to with HDMI.
The SPDF can't because it can't even read a 7.1 stream so it reverts to stereo 2.0.
Wrong again. Has nothing to do with SPDIF. Software configuration issue.
I guess I might be looking at a new receiver to achieve what I want.
That is of course your choice, but do it for the right reasons.