There are more photos here
that should give you an idea of the difference in glare/reflection.
The first image gives an extreme (ie, not likely to occur in most people's setups) example of the difference in reflections with and without the filter, with a brightly lit white panel held directly in front of the TV. The 2nd picture shows the difference in diffuse/indirect reflection (glare) from the screen.
The screen is definitely more reflective without the filter, because it's just plain glass. So yes, a little more attention needs to be given to the lighting arrangement and position of the TV, if you don't want to be looking at a mirror image of yourself whenever there's a dark scene.
The reflections don't really bother me though. The image looks clearer and sharper to my eyes without the filter, and shadow details still look inky black on the TV with proper room lighting.
All of those photos were taken a few years ago back when I removed the filter from my TV btw.
The white level pattern (shown above) was probably the best illustration of the difference in program content
that I was able to capture, given my limited skills with digital cameras at the time. I watched a number of DVDs with the filter left on just half of the screen though and there was a noticeable difference to my eyes, especially on brighter scenes. If you are a stickler for accurate color on your TV, you'll most likely want to recalibrate the grayscale, and probably also the black and white level settings on the TV if the filter is removed.