As afrogt mentioned, make sure that you've assigned to the optical input on the receiver to an input selector correctly. Make sure you don't have have an HDMI input assigned to the same input.
Make sure you've configured your PS3 to use the optical output as the primary sound output (Settings -> Sound Settings -> Audio Output Settings -> Optical Digital). Make sure that neither "AAC" nor "Linear PCM 2 ch. 176.4 KHz" are selected as output formats.
Your TV may not support sending audio to its optical output when you're playing movies from a USB device, especially if the audio isn't already encoded in a format that's compatible with SPDIF (optical).
If you're trying to connect your PS3 to the TV using HDMI and expecting the TV to output the HDMI audio it receives from the PS3 through it's optical out then it may not support this either.