Do you have any other devices successfully connected with both HDMI and optical?
Start by reading your manual, though granted, manuals from certain AVR manufacturers tend to suck, and read through this massive thread on your Denon AVR model:
http://www.avsforum.com/avs-vb/showthread.php?t=1152799
First figure out if you can assign optical 1 to HDMI 4, then how to tell it not to look for audio over HDMI, then how to define for that input what to do when it receives digital audio over optical, the setting you want is usually called something like "Auto Surround Mode." This is handled differently from brand to brand.
Do that first, then we can return to the Mac side of the equation if that's even necessary. You'll have to be more specific about 1) the software player and 2) the file you're trying to play back for us to help, but rule out basic Denon setup and configuration issues first.