In theory. Indeed that's what I had thought too and used SPDIF from a sound card for a couple of years. However with the card's native ASIO bit-perfect interface, it sounded great providing I didn't do anything on the PC while music was playing. Even just moving the mouse pointer caused dropouts and pops in the playback. I had to use universal ASIO4ALL instead for stable playback although it didn't sound as good but at least I could keep using my computer for other tasks and it didn't need to be just a dedicated music server. Kernel streaming or WASAPI didn't work very well with the card and the various media players I tried either. Jplay sounded good and gave reliable bit-perfect playback, but wasn't as user-friendly as a normal media player so didn't get used that much even though it had a very good sound.
These are common situations that can usually be resolved by adjusting some somewhat hidden parameters, upating drivers, moving other cards in the machine, etc. Sometimes they are not resolvable. In general, when they happen they are as you said unmistakable and non-subtle.
As I said above in post 56, using an analog interface is far from an instant kiss of death. It can sound as good as a digital interface. It can be sonically neutral.
However, you don't have to spend > $100 to obtain a good-sounding USB interface. Behringer's UCA 202 runs about $30 and both measures and sounds very good.
Analog interfaces get really awkward is multichannel, where you end up with a mass of analog cables.
There's a whole 'nuther path that is available to you if you switch over to HDMI. HDMI video interfaces run as little as $30 and can interface both audio and video well with AVRs. If you've got a modern desktop, there's probably an open slot for it.