...and somehow i think i have to rule out my receiver also, because
a) on the same input port on the receiver, i can hook up a dvd player with the same cable and it sounds flawlessly which means the receiver IS able to receive sound via toslink on this very port.
Most of what you'll pass from iTunes to an AVR is stereo PCM, whereas with a DVD player it's usually AC-3/DD 5.1 and DTS, but you're right, most AVRs by default should be able to receive either, determine what kind of stream it is, and switch back and forty accordingly. You didn't mention how far away the Mac is from the AVR or the length of the optical cable, so I wouldn't limit your hardware troubleshooting just yet...if it is a longer cable, like 25' to 50' then use a shorter one, say 3-6' and try the Mac going into a different optical input on the AVR, if not into a wholly different brand of AVR. And temporarily disconnect the FireWire audio interface as well (and any drivers you may have installed for it.)
I know you're resistant to trying a different AVR or lugging a Mac Pro over to a friend's house, do you have any friends with an Intel Mac laptop you could invite over for a quick test?
Couple other thoughts:
When you start playing a stereo audio file, something imported from a typical redbook cd as AAC or ALAC in iTunes like William and sivadselim asked over optical, does the track time counter start increasing or stay at 0:00?
Most of your testing so far has been with iTunes, right? So just for kicks, try playing a dvd of a commercial movie with Apple dvdplayer.app and switch between its different audio tracks...As you switch does your AVR react to the signal being stopped and started by blinking and then locking onto it, i.e. some recognition of say DD 2.0 versus DTS?