|Originally posted by csv96
I think double blind testing is great as long as the judges are experienced enough (especially to overcome their own biases) and as long as the goal or endpoint of the testing is clearly defined. The most interesting debate is the the Dolby Digital vs. DTS issue. The Dolby people seem to favor subjective evidence while the DTS people (i.e. referring to the DTS company) seem to favor objective evidence. I tend to favor the DTS argument, but who's to say that those who prefer the sound of a Dolby Digital soundtrack are "wrong"? Two questions could possibly lead to two completely different answers even with the same double blinded test. If the question is which soundtrack do consumers prefer and the double blinded test involved a Dolby Digital soundtrack vs. a DTS soundtrack, then if the judges are just common consumers there's a good chance the Dolby Digital soundtrack would win. Or if the judges are well trained audiophile movie enthusiasts, the DTS soundtrack might win ("more transparent", "better imaging"). On the other hand, if the question were which soundtrack sounds more like the original master and the double blinded test involved a Dolby Digital soundtrack, a DTS soundtrack, AND the original master, the DTS soundtrack might win even with the common consumer audience.
The correct way to do this sort of comparison is with reference to an uncompressed original. That is, you take various types of material and encode it in both formats. Then you carefully match levels, randomize trials, and so forth and determine how often experienced (and preferably trained) listeners with unimpaired hearing can actually tell the difference between the original and the compressed version and how they grade the severity of the degradation observed when there is an actual detection. At no time does anyone involved in the test know which is which, and it changes randomly from trial to trial. Good codecs are actually very hard to catch out; there are even libraries of sounds that are known to be difficult for perceptual coders that are used for evaluation by folks who are serious about this stuff.
This is really the only way to evaluate perceptual coders like Dolby Digital, DTS, MPEG, etc., because there is no set of measurements that can evaluate the masking effectiveness directly. People often fall back to arguing about data rate, but that's useless when trying to compare different codecs; it is very easy to design a poor codec that causes more impairment at a high data rate than a good codec at a much lower rate.
Finally, it's pretty much impossible to do a really meaningful comparison of Dolby Digital and DTS using commercially available material--partly because you don't have access to the originals and partly because the original may not even be the same for the two versions or there may be other differences in the processing applied. Too many uncontrolled variables. One, some, or even all people may prefer on over the other in any given case, but there's often no way to determine exactly why--and the one preferred may be the less accurate one.