I have had this happen a number of times in past years, where a "regular Joe or Jane" comes in and says "What did you do, it sounds better than before" and I haven't even mentioned having done anything. Same with my video picture. However, I guarentee that the straight objectivist will come up with reasons after reasons why such a test isn't a double blind or ABX test and therefore has no validity whatsoever. And to be honest, although I luv compliments on my theater sound, even if someone else thought my sound improved, but if my gut and ear/brain knew it became worse or not better despite spending a bunch of bucks, the end result most important is my perception since I'm the one who lives with it and my ear/brain and gut! HA!
-
Our native mobile app has a new name: Fora Communities. Learn more.
TRUE blind testing....
Having been a long-time lurker here and in other audio and video fora, I think this is an apt time to lose my AVS virginity, with a question that I don't think has been discussed here before in any organized way: Is "blind testing" (whether single or double) a misnomer as currently implemented, because the subject of the test is aware that a test is being conducted, hence is not blind to the actual test process itself (a problem compounded when there are multiple subjects in the same test session, as all sorts of human social interactions otherwise unrelated to the test can affect the outcomes)?
This question is spurred by my observation that, in any number of threads on AVS and elsewhere, I have seen many, many comments that changes in the performance of systems were noticed by persons who had no idea that any changes were made, persons who therefore were not expecting or looking/listening for differences, and yet the differences were significant enough to elicit unsolicited comments or questions. For example, a spouse comes home and says, "The sound/video is SO much better/worse -- did you do anything? What's different?" It seems to me that is is a TRUE blind test, as the subject is neither told that anything has been changed nor solicited for an opinion, and thus is reacting purely to a sensory input where no difference is expected and reacting to the memory of their previous experience of the system. On a personal level, I have had a number of such experiences, both as the owner of reasonably high-rez audio and video systems (having g'friends and friends with reasonable, but not, I think, extraordinary, auditory and visual acuity), and as the subject who himself noted differences in friends' systems, with no visual cues, such as a shiny new amp proudly on display, to clue me in.
This TRUE blind test would seem to address the so-called "objectivists'" claims that perceptions of changes to high-end systems effected by such things as power cords and conditioners, cables, similarly-spec'ed amplifiers and electronica, isolation devices, and other tweaks, are merely the result of the placebo effect, wherein differences are said to exist because we expect them to exist. Is there room in the "objectivist" paradigm for reports of this truly blind phenomenon as accurate data of the efficacy of the tweak or change that was done -- or will this merely be swept under the rug by calling it a lucky coin guess? I may put a Seismic Sink under my pre-amp, for example, and see or hear a difference, and the "objectivists" will invariably claim that, since there is no theoretical reason why air isolation will have any effect on an electronc device, my perception is merely wishful thinking or illusion -- but when someone with no knowledge that ANY change was made to my system comments on a difference (and particularly, as is the norm in my experience, where the comments on the differences are congruent with my sighted observations, e.g., more bass, solider colors, deeper sound-stage, etc.), isn't that rather conclusive evidence that the effect is objectively real and perceptible, even in cases where the effect is not measurable using common techniques and is thus denied by so-called "objectivists" as having any reality whatsoever? Any thoughts or comments on this observation? I can think of no explanation in the "objectivist" world view to account for this phenomenon, which I gather is quite common....
This question is spurred by my observation that, in any number of threads on AVS and elsewhere, I have seen many, many comments that changes in the performance of systems were noticed by persons who had no idea that any changes were made, persons who therefore were not expecting or looking/listening for differences, and yet the differences were significant enough to elicit unsolicited comments or questions. For example, a spouse comes home and says, "The sound/video is SO much better/worse -- did you do anything? What's different?" It seems to me that is is a TRUE blind test, as the subject is neither told that anything has been changed nor solicited for an opinion, and thus is reacting purely to a sensory input where no difference is expected and reacting to the memory of their previous experience of the system. On a personal level, I have had a number of such experiences, both as the owner of reasonably high-rez audio and video systems (having g'friends and friends with reasonable, but not, I think, extraordinary, auditory and visual acuity), and as the subject who himself noted differences in friends' systems, with no visual cues, such as a shiny new amp proudly on display, to clue me in.
This TRUE blind test would seem to address the so-called "objectivists'" claims that perceptions of changes to high-end systems effected by such things as power cords and conditioners, cables, similarly-spec'ed amplifiers and electronica, isolation devices, and other tweaks, are merely the result of the placebo effect, wherein differences are said to exist because we expect them to exist. Is there room in the "objectivist" paradigm for reports of this truly blind phenomenon as accurate data of the efficacy of the tweak or change that was done -- or will this merely be swept under the rug by calling it a lucky coin guess? I may put a Seismic Sink under my pre-amp, for example, and see or hear a difference, and the "objectivists" will invariably claim that, since there is no theoretical reason why air isolation will have any effect on an electronc device, my perception is merely wishful thinking or illusion -- but when someone with no knowledge that ANY change was made to my system comments on a difference (and particularly, as is the norm in my experience, where the comments on the differences are congruent with my sighted observations, e.g., more bass, solider colors, deeper sound-stage, etc.), isn't that rather conclusive evidence that the effect is objectively real and perceptible, even in cases where the effect is not measurable using common techniques and is thus denied by so-called "objectivists" as having any reality whatsoever? Any thoughts or comments on this observation? I can think of no explanation in the "objectivist" world view to account for this phenomenon, which I gather is quite common....
- Status
- Not open for further replies.
1 - 4 of 4 Posts
An interesting thought, but I'd think it would be somewhat hard to implement. Presumably in any test, blind or otherwise, you need more than one sample to have any claim on validity, and in such a case, it would be hard to pass off a deliberate deception in order to orchestrate the test. Moreover, with more than one person in the room, you would be subjecting the test to the social interactions that you were trying to eliminate - i.e., if one person says "it sounds muddier - doesn't it?", then a implanted suggestion would presumably bias the results.
FWIW, "blind" refers to the manner in which the test is conducted and the results interpreted. A singe-blind test is one in which the test subject does not know the identity of the two or more things to be compared, while a "double-blind" test refers to a trial in which neither the test subject or the conducter of the test knows the identity of the things to be compared.
I suppose I would fall into the "objectivist" camp, but I cannot recall when I've refuted a claim by analyzing the statistical validity of an ABX test that was conducted. In my experience, it's somewhat rare for any such test to have been conducted. More often, it boils down to a claim of "I can hear/see it" that is supported by an anecdotal report by one individual. These are interesting, IMO, as a basis for discussion, but not exactly useful in establishing whether the difference was illusory or the result of a physical change to the system.
FWIW, "blind" refers to the manner in which the test is conducted and the results interpreted. A singe-blind test is one in which the test subject does not know the identity of the two or more things to be compared, while a "double-blind" test refers to a trial in which neither the test subject or the conducter of the test knows the identity of the things to be compared.
I suppose I would fall into the "objectivist" camp, but I cannot recall when I've refuted a claim by analyzing the statistical validity of an ABX test that was conducted. In my experience, it's somewhat rare for any such test to have been conducted. More often, it boils down to a claim of "I can hear/see it" that is supported by an anecdotal report by one individual. These are interesting, IMO, as a basis for discussion, but not exactly useful in establishing whether the difference was illusory or the result of a physical change to the system.
I'm no expert in the area of double blind tests, but I'm surprised nobody has mentioned that ABX tests do not provide the best environment for determining if an audible difference between A and B exists.
The "A different than B" test provides the lowest threshold for statistically meaningful results in a double blind test. If all a listener has to do is determine if A is the same as B or different than B, that is a simpler test for the human subject than forcing him to identify if X is A or B.
I'm probably not giving credit where it is due, but many years ago an "A different than B" test was conducted by Stan Lipshitz et al. on transistor amplifiers. There was a broad mix of low distortion amps, from low end Japanese receivers to high-end American and British designs. All levels were carefully matched, no amplifier was allowed to clip, and listeners had as along as they wanted to listen BY THEMSELVES and conduct their listening tests. The results: nobody, not even the golden-eared listeners in the group could reliably identify any differences in sound quality amongst the amps under test.
The moral of the story: buy an amplifier for its power, reliability, and cost. It is also possible to identify amps by their noise floor signature by listening at close range to the speaker; the noisier amplifiers can stick out on sensitive speakers.
I'm sure I'll take some heat for supposedly trashing the heavy iron like Krell, Levinson, etc.
There's nothing wrong with buying beautiful works of art. Personally, I use Bryston amps. I think they represent a good value in both the high end and professional communities. Their 20 year warranty and customer service are absolutely first class.
-Dylan
The "A different than B" test provides the lowest threshold for statistically meaningful results in a double blind test. If all a listener has to do is determine if A is the same as B or different than B, that is a simpler test for the human subject than forcing him to identify if X is A or B.
I'm probably not giving credit where it is due, but many years ago an "A different than B" test was conducted by Stan Lipshitz et al. on transistor amplifiers. There was a broad mix of low distortion amps, from low end Japanese receivers to high-end American and British designs. All levels were carefully matched, no amplifier was allowed to clip, and listeners had as along as they wanted to listen BY THEMSELVES and conduct their listening tests. The results: nobody, not even the golden-eared listeners in the group could reliably identify any differences in sound quality amongst the amps under test.
The moral of the story: buy an amplifier for its power, reliability, and cost. It is also possible to identify amps by their noise floor signature by listening at close range to the speaker; the noisier amplifiers can stick out on sensitive speakers.
I'm sure I'll take some heat for supposedly trashing the heavy iron like Krell, Levinson, etc.
There's nothing wrong with buying beautiful works of art. Personally, I use Bryston amps. I think they represent a good value in both the high end and professional communities. Their 20 year warranty and customer service are absolutely first class.
-Dylan
1 - 4 of 4 Posts
- Status
- Not open for further replies.
AVS Forum
A forum community dedicated to home theater owners and enthusiasts. Come join the discussion about home audio/video, TVs, projectors, screens, receivers, speakers, projects, DIY’s, product reviews, accessories, classifieds, and more!
Full Forum Listing
Top Contributors this Month
View All
WilliamR
858 Replies
BASSAHOLIC
472 Replies
bass addict
396 Replies
Recommended Communities
