Consumers Union is the nonprofit organization that publishes Consumer Reports. The magazine is famous for the impartiality of its reviews, starting with the fact that the organization buys all the products it tests using secret shoppers. Naturally, the range of products the magazine tests includes televisions…many televisions.
I was unfamiliar with the current state of Consumer Reports' television-testing program, probably because it has been many years since I subscribed to the magazine. However, a chat with AVS member Jim Willcox (JakeNY28) revealed that he is actually the Senior Editor/Electronics at CR, and he graciously arranged for a visit to the testing lab.
I arrived at the 14-acre campus in Yonkers, NY, and met with Jim, who introduced me to Claudio Ciacci, the Electronics Testing Program Leader. We went straight to the lab and delved into details of the testing process, while I took in stark surroundings—Claudio designed the entire space to make testing televisions as efficient as possible.
Consumer Reports National Testing and Research Center in Yonkers, NY
Over the course of one year, Consumer Reports tests and reviews over 200 different TV models. Every unit goes through exactly the same battery of tests, all designed to reveal the full capabilities of each television. Nothing is taken for granted—even something as basic as whether a TV is 720p or 1080p is subject to independent verification—because printed specifications are not always accurate.
The lab is currently set up to test thirty TVs at one time
Claudio is responsible for how Consumer Reports tests televisions today. It takes five weeks for a TV to pass through the lab, and the lab tests 30 TVs in that time period. Technicians subject each unit to a series of tests that measure its capabilities while simultaneously identifying the optimum settings for that display.
The testing and review process is dauntingly technical. I saw the workbook in which all the collected data resides—it is amazing that Consumer Reports can distill so much information into its traditional method of ranking from poor to excellent. When one considers the explosion of features that are the hallmark of modern TVs, it becomes clear what a Herculean task it is to stay on top of it all.
The first thing the lab does is measure basic performance parameters—contrast, sharpness, color, resolution, native refresh rate, power consumption, etc.—as well as compile a list of features. The technicians go over each unit with a fine-tooth comb and essentially create a new spec sheet from scratch. Every part of the testing process is proprietary to Consumer Reports, including the test patterns.
A Photo Research SpectraScan PR-740 spectroradiometer takes hyper-accurate measurements
If a TV has premium features such as 3D capability, those also are tested. One of the more impressive examples revealed the amount of 3D ghosting generated by a given TV. I was bit shocked by the sorry state of active-shutter 3D technology, especially when it comes to LCD panels. For example, I did not know that crosstalk was such a big issue—thanks to Claudio's tests, its ubiquitous presence on active-shutter panels was obvious.
Claudio Ciacci explaining the testing process
Once I was in the lab, I began to grasp how serious Consumer Reports is about its TV-testing program. As Jim and Claudio explained it, in years past, the testing process took too long to be relevant, considering the frantic pace of development in the flat-panel arena. With traditional print publishing, it could take up to six months for a new TV model's review to appear—and even then, the sheer number of makes, models, and screen sizes made researching a TV purchase a daunting process. Fortunately, websites bypass all of these limitations, allowing for timely publication of reviews while incorporating search and sort tools that make comparisons fast and easy. This is crucial, because the pace of change in the TV industry is literally breakneck—it truly is survival of the fittest.
This year is a perfect example, with the introduction of OLED and the surprising growth of the UHD/4K segment. Screen sizes are going way up, and prices are dropping. Streaming and other "smart" features are standard on most TVs, and even "dumb" TVs are compatible with third-party streaming solutions such as ChromeCast, Roku, and Apple TV. Choosing the right model is more complex than ever before—a consumer still has to juggle many factors when making an educated decision about which TV to buy.
I asked Claudio about his opinions regarding video processing, such as noise reduction, so-called "vivid mode," and frame interpolation. He is totally opposed to gratuitous video processing, which came as a great relief. Part of the CR testing process includes confirming that the video-processing features—which are normally on by default—can be disabled by the user. This is not always the case with all TVs, especially when it comes to budget brands. Turning a feature off does not always completely disable processing.
Some of the test patterns and scenes used by Consumer Reports lab
Over the past year, I have had the pleasure of seeing the latest TVs, such as the $10,000 55" OLEDs from LG and Samsung, reference-quality plasmas from Samsung and Panasonic, and some amazing full-array local-dimming UHD/4K LCD TVs such as Samsung's 84" S9. At the extreme high end, image quality is very close to perfect. However, the vast majority of TVs sold today are LED-edgelit LCD panels, which suffer significant image-quality issues—something I've seen demonstrated numerous times in controlled environments.
Even so, it's always a shock to see tests designed to show the flaws with LED-edgelit LCDs. Simply put, the same technology that allows TVs to be thin, light, and energy-efficient also results in inferior performance—largely characterized by uneven illumination—that can be extremely distracting.
Edge lit LCD TVs suffer from halo artifacts when rendering bright objects on dark backgrounds
One of the more punishing tests for edgelit LCD TVs is watching a simple white object traveling horizontally across a black background. Manufacturers often claim some form of "local dimming" for these units, but the simple fact is there's no way to isolate the light coming from the edges to illuminate a bright object in the center of the screen. An artifact known as flashlighting is clearly visible, as is an amorphous bright blob around the white object—a halo artifact. In the same room, a calibrated high-end plasma television served as a reference unit. The white object tracked perfectly on the plasma, with no halo or flashlight artifact. The combined difference in black levels and lack of artifacts demonstrated a stunning gap in performance between the two technologies. Plasma fans certainly know what they are talking about when it comes to image quality issues and edgelit LCD.
There are two basic types of LCD panels—VA (vertical alignment) and IPS (in-plane switching). Each has its advantages and disadvantages. When viewed head-on, VA panels tend to have deeper blacks than IPS, but when viewed from the side, IPS panels maintain picture quality while VA panels tend to lose saturation and contrast. I knew about these characteristics already, but Claudio showed me something I did not know about—when viewed from below, many IPS panels suffer the same loss of contrast and saturation that VA panels do when viewed from the side.
A little light bulb lit up above my head, and I asked Claudio, "Aren’t many TVs mounted above a fireplace mantle? This could be a real issue with IPS panels." He agreed, saying that buyers of IPS panels should make sure to get a wall mount that tilts in that case.
The point is that TVs present a complex subject to cover. There are many makes and models, and finding the right one for a specific application takes a fair bit of research. Most of this research is impossible to perform in a store. In fact—and this might be a bit controversial—I think that many people would actually be better off shopping for a television using nothing but online tools, as opposed to trying to pick out the right television while standing in a big-box discount store. I am not referring to high-end retailers, where there is a light-controlled environment and a knowledgeable salesperson—I'm talking about the giant wall of TVs where many shoppers choose their flat panels.
For example, when it comes to discount televisions, manufacturers sometimes use panels from different suppliers in different-sized televisions within the same line. As a result, performance is not consistent throughout all screen sizes, which is why it is necessary to test all the different screen sizes. Another factor is firmware version, because sometimes a firmware update can fundamentally change the settings that are optimal for a given television.
The point of the CR lab's process is to get the televisions to the best possible state in terms of picture quality, short of a professional calibration. As Claudio explained it, if a television really does require professional calibration to look great, that will actually count against it. The reason is simple—many modern televisions actually do perform very well out of the box. Of course, it also counts against a TV if standard tweaking does not result in a good picture.
By design, the tests are as objective as possible; any trained technician can take the place of any other trained technician and produce essentially the same results. The technicians log the optimum settings, along with the accompanying firmware version. At this point, the TV moves on to the final testing phase.
TVs lined up for testing
As I mentioned earlier, there is a reference-quality plasma television in the lab—a fully calibrated Panasonic TC-P55VT50. Actually, there are two of them. Each television has to go up against this reference, which is a tough TV to beat. Then, the techs watch a variety of content on the optimized TV in order to get a handle on how it performs an actual use, as opposed to how it measures. Ultimately, both factors are relevant to any thorough review.
Of course, there is a catch—what Consumer Reports does is undoubtedly an expensive endeavor. As I noted at the beginning, the nonprofit Consumers Union publishes the magazine, which eschews advertising and does not accept product samples from manufacturers. That leaves one source of funding—paying subscribers. The magazine itself requires a subscription, of course. However, when it comes to television reviews, consumerreports.org offers the most comprehensive resource for researching a potential purchase. Some access is free, but access to ratings, comparison tools, settings used to optimize televisions, and other premium features require a subscription.
I cannot help but be impressed by the program Claudio and Consumer Reports have put together. As Jim described it, prior to Claudio, the publication's television reviews were relatively subjective. Claudio injected more science and objectivity into the process and added economies of scale. The result is an unprecedented amount of TV testing—at the current pace, Consumer Reports will test 300 different units each year. I do not know of any other outlet that is testing that many televisions as thoroughly and scientifically. I want to thank Jim Willcox for making my visit possible, and Claudio Ciacci for taking the time to show me the lab. I also want to thank engineers Christopher Andrade and Matt Ferretti, who were present on the tour and do a lot of the heavy lifting in the lab.
Consumer Reports has a lot of experience with testing TVs
Finally, I would like to note that AVS Forum features a user reviews section, which can offer insights into how actual consumers perceive products. Proper research includes looking at a mix of professional reviews, user reviews, and actual TVs. The average American watches around four hours of TV per day—far more time than they spend driving a car. Clearly, it is wise to learn as much as possible about all the available options before walking into a store.
Edited by imagic - 10/23/13 at 11:02am