The Limited/Full setting has no effect when the PS3 is outputting YCbCr, just like the SuperWhite setting has no effect when the PS3 is outputting RGB.
The recommended settings (RGB Limited and SuperWhite ON) will each take effect only when the PS3 is sending out RGB or YCbCr respectively.
For most people, Blu-Ray movies will be output by the PS3 as YCbCr, and thus the RGB Limited/Full selection makes no difference.
However, some people have displays which insist on receiving RGB -- displays with DVI inputs being the usual culprits. And some games insist on RGB output. For those cases the RGB Limited/Full selection does make a difference (and the SuperWhite setting is ignored).
The key thing to remember when the PS3 (or any other device) is sending out RGB, is that the device at the other end of the cable has to be set to expect the same flavor of RGB.
For home theater use, what the PS3 calls RGB Limited and what everyone else calls "Studio RGB" is the proper default choice. There should be a corresponding setting in your TV or receiver -- and again it should only have effect when the PS3 is actually sending out RGB. The setting in the TV or receiver could be called just about anything as the marketing guys who come up with these names like to invent new names, but it will have just two choices. It will most likely be called something like Blacks with the choices being something like Lighter/Darker, Normal/Extended, 16/0, or perhaps something even more confusing.
There are a few displays and projectors out there which are designed primarily for use with computers that insist on RGB input *AND* do not have enough calibration range to handle RGB Limited (Studio RGB) input correctly.
And that's why the PS3 offers the RGB Full choice. If you find you can't make RGB Limited work with your TV or receiver when the PS3 sends out RGB, then switch the PS3 to RGB Full and try again.
Note that despite the way these things are named, they really have nothing to do with making the picture lighter/darker or even better/worse. The only reason you see a difference when you make the change in the PS3 is because you have not ALSO made the corresponding change in the TV or receiver.
RGB Limited (Studio RGB) encodes Black as digital 16 and Reference White as 235. The ranges from 0-15 and from 235 to 255 are used in video processing to hold the so-called Blacker than Black and Peak White data.
[NOTE: There is a long standing bug in the PS3 that prevents it from actually sending out Blacker than Black or Peak White data when RGB output is used. That's one reason why YCbCr output (with SuperWhite ON) is the preferred choice for Blu-Ray movie watching.]
RGB Full (Extended RGB) encodes Black as digital 0 and Reference White as 255. Since no pixel can have a value below 0 or above 255 there is no place to put Blacker than Black or Peak White data.
Now if the PS3 is set to send RGB Limited, and the TV or receiver is set to expect Black=0, then the image is going to look a bit washed out in the near blacks. You would have to compensate by lowering black levels (Brightness control) in the TV or receiver.
Similarly if the PS3 is set to send RGB Full, and the TV or receiver is set to expect Black=16, then the image is going to look too dark and with loss of near black details. You would have to compensate by raising black levels (Brightness control) in the TV or receiver.
But if both sides of the cable are set the same way then blacks will look correct USING EITHER SETTING.
Anyway, here's the formula:
1) Set the PS3 to RGB Limited and to SuperWhite ON. Set the PS3's output choice to Automatic (rather than either YCbCr or RGB).
2) Set the TV or receiver to expect Studio RGB whenever it realizes the signal coming in is RGB. Studio RGB is the one where Black=16
3) Using a Blu-Ray calibration disc (such as Digital Video Essentials Blu-Ray), set the basic levels in your TV or receiver. If you have no problem doing that then you are done.
4) But if you find you can't set Blacks properly, and if you discover the PS3 is sending RGB (perhaps because you have a DVI input display), then, and only then, try switching the PS3 to RGB Full output.