Originally Posted by Chu Gai
If the protocol has been spelled out as to what tests are going to be peformed and every sub is evaluated by those criteria, it's irrelevant. Same goes for evaluating a product's strengths and weaknesses when it comes to construction, fit and finish. Photographs can accompany these sorts of comments to substantiate the assessment. Commenting on the subjective sound quality is of course open to debate but that holds true regardless of where or who does the evaluating. I've found that AH hasn't always been effusive in its praise for products it also sells through their retail channel. As for the costs that a vendor has to incur in paying the s/h charges back and forth, that's just written off. I kind of suspect that some, but not all, vendors don't want their subs evaluated because they just might get called out on some of the claims they've made and really aren't that confident in the performance or construction of their product.
I agree, for the most part. But that fine line between what is "best" and "best value" are very subjective. One sub could beat another, but it might cost $500 more. If the Klipsch sub cost more than Brand X the final result might be "Clearly the Klipsch is the Ultimate Sub"
. If it was reversed the final result might be "Brand X sub won all the performance categories, but for the $500 you could pocket by getting the Klipsch sub, you'd be better off using that to buy [fill in blank]."
Then we get into the more ambiguous areas like does it matter if one sub can go to 5 hz while another can only go to 10 hz, but the 10 hz sub just sounded tighter and more refined.
IMO, there are too many variables for a fair review when you factor in the web site is selling some of the brands in the test. Considering they rec'd Yamaha receivers for nearly all their rec'd systems, I find it hard to believe the separation between the store and the reviews. But I'm skeptical by nature.
PS: Chu - if Schifter was holding this event and published the protocol, I'd imagine you'd want a representative from each company present to observe the results.