Originally Posted by andy sullivan
...They blind test PQ using techs and non techs, they add up the results and publish them....
I think this gets to the heart of the matter.
By including a diverse group of reviewers, with different levels of experience and sophistication, CR employs more of a "focus group" approach.
I do have to deal with focus groups often, and I generally dislike them. They serve to cover the behinds of marketing departments, who in the case of a product failure, can always point to the record of the focus group, and claim that nobody expected the failure, thus nobody should be blamed for it:-)
One problems with focus groups is, that often they employ poor methodology. For instance, if CS had 5 testers in the same room, the group opinion can easily be colored by one "leader" participant.
Another problem is, that focus groups tend to smother vision and bring everything to the lowest common denominator. If you asked the crowd at Dairy Queen to pass on the merits of say, art, you may very well end up with a velvet painting, rather than a Van Gogh....
And they are often wrong. For instance, if I remember correctly, Ford famously decided that there was absolutely no US market interest in a roadster, after extensive focus-group testing. That was just a couple of years before the introduction of the Miata, which of course started a decade-long roadster-fad. Similarly, the Prius tested horribly in focus groups.
Anyway, while this way of testing may have some merit, it also results in the jacked-up TV settings one sees on the floors of discount retailers (and in many homes:-)
So, there is something to be said for individual reviewers, whom you can judge based on your personal agreement or disagreement with their past reviews.