I hooked up my HTPC's digital coax output to my JVC A/V receiver digital coax input and tried playing a CD in MythTV, but get no sound. I haven't bought a digital coax cable yet - is it really different than a video coax or RCA cable? Just for the quick test, I used an RCA cable. I could try using a spare video component cable also - doesn't look much different than the pictures of a digital coax cable. My JVC manual isn't very clear as to how to use the digital coax. There is a digital input column on the back of the receiver with a coax port and 2 optical ports below that. On the front panel, one is supposed to cycle through some input options - I selected 1-CD, 2-DBS, 3-DVD, and tried setting my source input to CD-Digital and DVD-Digital with no sound from either.

I haven't tried playing a DVD on my HTPC yet, and currently have problems getting my HDHomeRun TV tuners to work with OTA and QAM so can't test those yet.

I haven't bought a digital coax cable yet - is it really different than a video coax or RCA cable?

No there is no difference, copper is copper is copper, don't fall for the Monster Cable crap.

I use a JVC receiver (RX-888) as well, and it sounds like your digital inputs are close to mine. Just set the coax port for any source and then make sure that you have that source selected. For example I set my coax digital input for DVD and when I want to watch movies I select DVD-Digital on my receiver.

It sounds like you are setting up your JVC correctly though. My guess is that is probably the way the ALSA driver is set up, it tends to default to the analog outputs. From a shell prompt run alsamixer or if you run a KDE environment try running kmix. Of course a lot depends on your your linux distro was installaed in the first place, but most recent distro's now default to ALSA.

I do use Myth, I use Freevo instead, so I can't say if Myth has it's own audio requirements that need to be setup.

You need to tell Myth to use the spdif. In sound settings try the other settings like '/dev/adsp' for oss or alsa:spdif or alsa:iec958...There are likely more depending on what your setup is so you'll have to check them out. If you can't get it going, google mythtv spdif and you will get tons of info.
-Trouble