Well, don't forget, it all adds up. There is input lag on the wireless controller to most of the current consoles of a few milliseconds. Then there's the lag of when you hit a button on the controller to when the frame associated with that press of the button is generated at the console, which can be anywhere from about 30 to about 100 milliseconds (it tends to be much less on a good PC though). Then there's the input lag of your TV (which are listed in this thread). They can all add up to being noticeable.
Then there's also the situation where you see an enemy in a game, you aim at him, you pull the trigger, and then you die without shooting. You sit there, dead, thinking "man, I swear I pulled the trigger before he shot me!" In a situation like that, you might notice even very low input lag.