x264/h264 avc codec is fine for video. I always keep the original AC3 Dolby Digital audio.
With the PS3, the only format with switchable subtitles is AVI container with a DIVX/Xvid encode.
For MP4 you need to hardcode burnin the subtitles, this option is in Handbrake.
PS3 will happily handle High Profile, level 4.1, i set CRF to 16.5, I tune the default video settings so that B-Frames is set to optimal and motion search is set to uneven multi hexagon. I leave all the other settings.
Due to the subtitle issue and iphone/ipad compatibility. I have moved from using the PS3 to AppleTV for streaming video. The AppleTV/ipad/iphone can all have switchable soft subtitles in the mp4/m4v container.
I had been using AVI / Xvid with the PS3 up till last year. But x264 is such a better encode that I re-did my entire library in the new config. 720p/1080p xvid does not stream well to the ipad.