The optical/coax digital audio outputs will carry undecoded, multi-channel, digital audio from the player to the receiver. The receiver will decode it, process it according to the various features of the receiver, convert it to analog, and amplify it before sending it to your speakers.
Alternatively you can use the 5.1 ANALOG audio outputs of your player. In that case the player decodes the signal and processes it according to the features of the player, then converts it to analog and sends THAT to the receiver. Typically the receiver will then act as just an amplifier, providing volume control, although some receivers offer the option to process analog audio inputs before sending them out to the speakers -- usually by converting them back to digital signals first.
The sort of processing involved here includes bass steering (some bass from each channel diverted to the subwoofer instead), volume balancing between the speakers, time-alignment (adjusting for the different distance of each speaker from your ear), and, possibly, frequency based room equalization to correct for room problems. Depending upon whether the player or the receiver does a better job of these tasks, as well as the most basic but also the most vital task of converting the digital signal into analog format, you might prefer to have the work done in the player or in the receiver. Try both and see for yourself.
There's one other important gotcha here. The exotic audio formats -- DVD-Audio and SACD -- produce higher bandwidth multi-channel audio than can be carried by the optical/coax digital audio connector. Thus to listen to those properly you will HAVE TO hook up the 6 analog audio cables from the 5.1 analog outputs of the player to the receiver so that the player can decode and convert the exotic audio properly.
EXCEPTION: Some players and receivers have a DIFFERENT style of digital audio connection called I-link (IEEE1394 or "firewire") which IS capable of carrying the exotic audio formats as well as everything else. If you have I-link equipped stuff, then once again you have the choice of having the player or the receiver do the work. [NOTE: I-link and Denon-Link are NOT the same thing.]
Note that some players do a GREAT job of handling DVD-Audio and SACD but not so great a job for the more mundane formats of CD and DVD. Thus it is rather common for people to hook up *BOTH* an optical/coax digital audio connection *AND* the 5.1 channel analog audio cables and switch between them according to what they are listening to. Note that if you, too, do that, you need to setup speakers and such in BOTH the receiver (for when it is doing the work) AND in the player (for when IT is doing the work).