I am actually working on this right now as part of my house wiring.
Things to think about are:
Controller ranges- I've looked and there aren't a lot of ways to extend the ranges.
EDID- If you are using a cheap matrix you will likely be limited to the highest common denominator of your displays. If one of your Displays is 720p, that is likely the best you will be able to get. To get around this I purchased 3 identical Onkyo receivers.
Kinect/Move - They don't use...