Hate to break it to you, but almost all TVs only output 2-ch audio from the optical out for everything but OTA signals, so even if your set-up was working, you wouldn't be getting 5.1 sound. If you want 5.1 channel sound, you need to run the optical between the Xbox and the reciever, not the TV and the reciever.
How are you getting your TV signal? If it's through cable, then you will have to figure out another way to set things up in order to get 5.1 channel sound, because I can almost guarantee that you are not getting it from you optical out on the TV. The optical out on TVs is intended for when you are using the TVs tuner, not as a pass-through for another source.