For watching video, ignore it. It doesn't matter. It really only matters for gaming, and even then a lot of people claim they don't notice it. You're most likely to see/feel it on 2D/retro games, fighting games, using a mouse/keyboard, etc. Console games or using a joystick, it's very likely you won't notice.
As far as what amount is "horrible," it really depends on the person. I bought the ST50 because I'm sensitive to the lag and it's in the 70-80ms range (IIRC). Game mode helped, and I believe brings it down to 40-50ms. However, my receiver was introducing input lag as well... So now my console/PC is directly connected to the TV, and game mode is on, which should put the total lag at 40-50ms, and I can't notice it without really trying.