Chances are your TV will not take an HDMI 5.1 and output the 5.1 to your receiver. It will output 5.1 on HD channels but it can't pass through DD.
For xbox, there is absolutely no difference in quality between HDMI and optical cable. Xbox is only capable of 5.1 Dolby Digital. The reason you hear HDMI is better is because it is capable of handling 7.1 channels lossless audio. This includes Dolby TrueHD, DTS-MA, and 8 Channel PCM. Dolby Digital is compressed audio. For your situation, run HDMI to TV and use optical cable to your receiver.
Also, unless your receiver is capable of HDMI input, you are not able to get 7.1 sound out of your PS3. Optical straight to the receiver is the best option you have.