DTS or PCM Surround? NAD 777 + PS3

I doubt that you could ever hear the difference. Basically you are just choosing whether you want the PS3 to decode the PCM or the 777. A nice receiver like the 777 will do a better job because it has a more sophisticated decoder. So just let the PS3 pass on the raw signal and let the 777 decode it.

"A nice receiver like the 777 will do a better job because it has a more sophisticated decoder."

No. Decoding a compressed digital stream results in the same bits no matter how fancy the device is doing the decoding. There is no guesswork involved and no "doing it better".

Decoding at the PS3 lets the PS3 mix menu audio and special feature audio in to the stream before output. This is a good thing, the correct thing for the Bluray standard, and the intention for HDMI audio transmission. Every device should decode at the source and spit out LPCM over HDMI. Bitstreaming of DD/dts over HDMI should not exist and only exists because marketing got a hold of the standard and did a great job of convincing users that the poorer solution was a great idea.

The PS3 is an excellent BluRay/CD/DVD player.

Auto works just fine 99.9% of the time.

"until I get a decent Blu-Ray / CD player"

Aside from the lack of native IR remote support, it's pretty difficult to get a better Bluray, film DVD or CD player than the PS3. Its weak point is 60hz TV on DVD, as deinterlacing 60hz 480i is still pretty easy to get wrong. Really, any TV you'd watch on DVD is on your Netflix or Hulu account in HD anyway. The PS3 is one of the earliest Bluray players, but because of its massive install base it has had by far the longest running and best support in terms of feature updates. It will take something major like official support for 4K or 48hz 1080p 3D to make your PS3 obsolete.

"Is this ideal or is there a better way to set the Audio output from the PS3 that will get the most out of this receiver?"

Auto means your receiver reports to the PS3 what it supports, then the PS3 sets it's output options accordingly. Auto is the correct choice, unless your receiver is broken and doesn't correctly report what it does.

Auto is fine.
darklordjaes does have a point about the ps3 mixing the menu audio and special feature audio before output. Thats why if you choose bitstream it says "some portons of the audio MAY not be played". That is the portions that it is talking about. But I have never had anything not play.
Again this is really insignificant and nothing to get workd up about. I prefer my receiver to do the decoding. But I wanted to point that out in case you ever do run into some audio that does not play.
The ps3 isn't the best player on the market but it does fine. It would not be worth getting a dedicated player as long as you have the ps3 imo unless you are looking for some special features that the ps3 does not have.