Most everything is based on RGB, and somewhere along the line are decoders and transcoders (conversion) involved.
I would suggest you start here to better understand about RGB and YCbCr .
You may even want to check out your HDMI cable, you would want a high speed 1.3 HDMI cable.
You also said something about a AVR in your chain, if you are running your Roku through your AVR look there also.
Also check out any Roku threads for issues.
The other things you could try with your VT60 is.
Turn Off 1080 Pure Direct.
HDMI content Type Off
Auto detail settings, Cinema Off
Try a different HDMI port in your VT60.
Do a factory reset.
When you say you switch to your Roku your backs get messed up, leads me to believe there is a miss communication somewhere in your chain.
There are many combinations (settings) in a video chain involving AVR, Roku, PS3, cable box, Blu Ray players, ect.
If none of the above suggestions help, I would run each device directly to your VT60 and see if there is any problems. At-least that way you can pin point the problem and go from there.