I assume you are talking about the main audio settings, and not the menu that you can access when playing blu-rays.
This setting will determine how game audio is encoded and sent to your receiver or TV. The PS4 will be mixing audio on the fly according to what is happening in the game, and will not really be "decoding" anything. The PCM setting will send the raw audio. (7.1 over HDMI or 2.0 over optical.) If you have this set to Dolby Digital or DTS, it will encode the audio into that format and bit-stream it out for your receiver to decode. This is on both the HDMI and optical outputs.
If you are using optical out, it is most likely because your AVR does not have HDMI inputs. In this case, you have a real choice to make, lossless 2.0 audio (PCM), or lossy 5.1 audio* (Dolby Digital and DTS). I'd go with the lossy 5.1 option (DTS if your AVR supports it.)
Hopefully you are hooking up to your AVR via HDMI. In that case, leave the game audio set to PCM. With the Dolby and DTS options, you will be getting lossy audio limited to 5.1, and adding an encoding step on the PS4 and decoding step on the AVR. The PCM is lossless and, according to my AVR, 7.1.
Hope this helps.
*This may be 5.1 EX or 6.1 DTS ES, but I'm too lazy to check right now. Either way, the gaming DTS and Dolby Digital will be the older lossy variants, not lossless DTS-MA or Dolby TrueHD.