Mac Mini - DVD player sound output

Given the way you have your Mac connected in the DVD player app preferences you should be seeing two audio output choices, "system sound output" and "digital out." The one you want is digital out. (If you had two digital out connections that could carry audio, as one might with a newer Mac than yours, say a Mac with HDMI out and optical out connected at the same time, you'd have to choose which digital out you wanted, and in that case for optical you'd choose "built-in output." But you don't have to make that choice, just toggle from System Sound to Digital Out and you should be fine.

In the System wide Sound output preferences panel, the device you should select is "Digital Out" and "optical digital-out port."

If you do all that, and still aren't getting any sound, then it's human error setting up your AVR.

Also, can you try to play back a ripped dvd to confirm you can pass-through DTS or DD 5.1 with a different program, say VLC or XBMC?