Joined
·
18,200 Posts
[Note: This is reposted from another thread on the "audible" differences between USB cables.] It is more pertinent to this section of our forum.
Despite being called the "gold standard" of the prestigious scholarly science Journal of the Audio Engineering Society, as one published scientist recently put it, many people pushing audio mythology (usually for profit) can't stand double blind tests such as ABX because it exposes their falsehoods. Here's an example I dissect.
Jump to 47 seconds in (if my link below doesn't take you there directly):
Paul, at 47 seconds in: ". . . and I don't remember what we were trying to test for . . . "
https://youtu.be/3mHaSb8Nc8c?t=47
So for all we know he was testing for example speakers, or any number of other items where we should expect listeners to be able to hear a difference.
He also lies, or perhaps just incorrectly remembers, about how it works:
"You had a button and you could hear
A [*uses hand gesture to show a button being pressed*],
B [*uses hand gesture to show the exact same, single button being pressed*]
or X [*uses hand gesture to show the exact same, single button being pressed*]
WRONG Paul. You have multiple direct access buttons on the hand held remote so you can jump straight to the sound you want, A, B, or X, instantly and directly, at any time you choose. It is not a sequential thing, as you imply, trying to make it seem like a much more difficult test which requires memory.
Here's the remote he used:
http://djcarlst.provide.net/abxhcm.gif
Deceptive people do their best to fool people into thinking the way Clark's ABX switch box worked involved having to juggle in one's mind three different sounds, all at once, in memory. THIS IS PATENTLY FALSE. ABX is an open book test so the sounds of A and B, fully labelled and identified for what they truly are, require no long term memory at all. The only sound with a hidden identity is X, which is either A or B. [The up down buttons on the remote allow the listener to scroll up and down between what trial number the listener is voting on, as displayed on an LED readout they look at during the test, by the way.]
". . . and you didn't know which was what. . . "
FALSE. Great job wording it to scare people so they think they are juggling three unknowns in their mind though, Paul. The listener knows exactly what A is, what B is, [and if the test designer chooses, their exact brand, model number, appearance, and price] and can instantly re-hear them as many times as they want, both before and after they hear the mystery sound X as many times as they want, which they are voting on as being A or B.
[*waves hand around to show a state of confusion*] "so the testing people didn't know, you didn't know. And. Um. There were a whole bunch of things wrong with that."
And all these problems [none of which Paul seems to have explained yet] evaporate away by making the test sighted, Paul? NOPE.
Later he mentions the "sound degradation of switching relays".
A. For some reason the input selector relay in his own company's integrated amps somehow manage to be beyond reproach.
B. You can run a double blind pre-test, of just the switch box against straight wire, to see if the switch/relay indeed causes "audible degradation" as he fears. [It doesn't.]
C. The music we listen to has passed through hundreds if not thousands of various switches/relays along its journey from the recording studio, to the mixing, mastering, etc.. before it reaches our ears and nobody seems to complain about those.
D. When I run blind tests, as I explained in my video, I give the listener the option to use hard wire cable swapping instead of a switch. Their choice. [After all, when they "heard a difference" between DACs, USB cords, whatever, wasn't cable swapping exactly how they were doing it?]
"The resolution of the speakers was grim."
[My listeners use whatever speakers, and other peripheral gear they want. I don't choose them, they do.]
"I think they were Snell?"
Interesting he can remember what the speakers were but not what was being tested! [Psst, I bet he picked Snell for his little story as a back handed slam against Meyer and Moran's testing, if you ask me.]
". . . and undisclosed electronics."
In all blind testing, including ABX tests, all the electronics are fully disclosed to the listeners in any I have ever conducted or read about. But again, he is trying to scare people away from even trying it.
"It certainly wasn't setup properly for imaging, spatial cues, and the things I am more sensitive to. . . ." [brag]
I can't speak for everyone, but in my blind amp test the entire setup was done by the recording engineer I tested himself, using his gear of choice, his treated room of choice, his preferred speaker placement and positioning, and his music of choice.
"If we're not in the right frame of mind we aren't going to hear. . . "
So I guess this problem you speak of, Paul, doesn't occur with sighted tests because people are always in the right frame of mind when conducting sighted tests. Riiiiiiiiiiiight. Uri Geller uses the same deception to pass off his failures: "I'm not feeling strong tonight" he says to Johnny Carson in the earlier video. Fine. We'll stop the test and and start a new one whenever you feel you can make decisions about sound using only your hearing, not your sight, Paul.
"If I'm on public display, if there's pressure on me, I clam up. I don't do well at all. If I have an audience and I'm being judged by that audience. My brain just kinda turns off.. . . If I'm being tested I don't do well."
He means in a double blind test he doesn't do well. In sighted testing he brags earlier that he has a discerning ear. Also, when doing a Foobar ABX test nobody is in the room but you, there is no audience during the test, and you can take as long as you want, even pausing the test one day and starting back up hours, days, weeks, or months later, whenever you are feeling in the mood to hear differences using your ears, not eyes.
He ends by mentioning he likes "blind" A/B testing, switching back and forth quickly between A and B, (with a helper in the room). He's done a splendid job in this video of fooling people into thinking this exact same technique of quickly switching from A to B can not occur in ABX testing, whereas in truth many users doing ABX choose to do exactly this. I use this method quite often.
Despite being called the "gold standard" of the prestigious scholarly science Journal of the Audio Engineering Society, as one published scientist recently put it, many people pushing audio mythology (usually for profit) can't stand double blind tests such as ABX because it exposes their falsehoods. Here's an example I dissect.
Paul doesn't know what he is talking about, makes numerous factual errors and misrepresentations, and admits that in the case of his ABX test, 30 years ago, he can't even remember what was being tested:
Jump to 47 seconds in (if my link below doesn't take you there directly):
Paul, at 47 seconds in: ". . . and I don't remember what we were trying to test for . . . "
https://youtu.be/3mHaSb8Nc8c?t=47
So for all we know he was testing for example speakers, or any number of other items where we should expect listeners to be able to hear a difference.
He also lies, or perhaps just incorrectly remembers, about how it works:
"You had a button and you could hear
A [*uses hand gesture to show a button being pressed*],
B [*uses hand gesture to show the exact same, single button being pressed*]
or X [*uses hand gesture to show the exact same, single button being pressed*]
WRONG Paul. You have multiple direct access buttons on the hand held remote so you can jump straight to the sound you want, A, B, or X, instantly and directly, at any time you choose. It is not a sequential thing, as you imply, trying to make it seem like a much more difficult test which requires memory.
Here's the remote he used:
http://djcarlst.provide.net/abxhcm.gif
Deceptive people do their best to fool people into thinking the way Clark's ABX switch box worked involved having to juggle in one's mind three different sounds, all at once, in memory. THIS IS PATENTLY FALSE. ABX is an open book test so the sounds of A and B, fully labelled and identified for what they truly are, require no long term memory at all. The only sound with a hidden identity is X, which is either A or B. [The up down buttons on the remote allow the listener to scroll up and down between what trial number the listener is voting on, as displayed on an LED readout they look at during the test, by the way.]
". . . and you didn't know which was what. . . "
FALSE. Great job wording it to scare people so they think they are juggling three unknowns in their mind though, Paul. The listener knows exactly what A is, what B is, [and if the test designer chooses, their exact brand, model number, appearance, and price] and can instantly re-hear them as many times as they want, both before and after they hear the mystery sound X as many times as they want, which they are voting on as being A or B.
[*waves hand around to show a state of confusion*] "so the testing people didn't know, you didn't know. And. Um. There were a whole bunch of things wrong with that."
And all these problems [none of which Paul seems to have explained yet] evaporate away by making the test sighted, Paul? NOPE.
Later he mentions the "sound degradation of switching relays".
A. For some reason the input selector relay in his own company's integrated amps somehow manage to be beyond reproach.
B. You can run a double blind pre-test, of just the switch box against straight wire, to see if the switch/relay indeed causes "audible degradation" as he fears. [It doesn't.]
C. The music we listen to has passed through hundreds if not thousands of various switches/relays along its journey from the recording studio, to the mixing, mastering, etc.. before it reaches our ears and nobody seems to complain about those.
D. When I run blind tests, as I explained in my video, I give the listener the option to use hard wire cable swapping instead of a switch. Their choice. [After all, when they "heard a difference" between DACs, USB cords, whatever, wasn't cable swapping exactly how they were doing it?]
"The resolution of the speakers was grim."
[My listeners use whatever speakers, and other peripheral gear they want. I don't choose them, they do.]
"I think they were Snell?"
Interesting he can remember what the speakers were but not what was being tested! [Psst, I bet he picked Snell for his little story as a back handed slam against Meyer and Moran's testing, if you ask me.]
". . . and undisclosed electronics."
In all blind testing, including ABX tests, all the electronics are fully disclosed to the listeners in any I have ever conducted or read about. But again, he is trying to scare people away from even trying it.
"It certainly wasn't setup properly for imaging, spatial cues, and the things I am more sensitive to. . . ." [brag]
I can't speak for everyone, but in my blind amp test the entire setup was done by the recording engineer I tested himself, using his gear of choice, his treated room of choice, his preferred speaker placement and positioning, and his music of choice.
"If we're not in the right frame of mind we aren't going to hear. . . "
So I guess this problem you speak of, Paul, doesn't occur with sighted tests because people are always in the right frame of mind when conducting sighted tests. Riiiiiiiiiiiight. Uri Geller uses the same deception to pass off his failures: "I'm not feeling strong tonight" he says to Johnny Carson in the earlier video. Fine. We'll stop the test and and start a new one whenever you feel you can make decisions about sound using only your hearing, not your sight, Paul.
"If I'm on public display, if there's pressure on me, I clam up. I don't do well at all. If I have an audience and I'm being judged by that audience. My brain just kinda turns off.. . . If I'm being tested I don't do well."
He means in a double blind test he doesn't do well. In sighted testing he brags earlier that he has a discerning ear. Also, when doing a Foobar ABX test nobody is in the room but you, there is no audience during the test, and you can take as long as you want, even pausing the test one day and starting back up hours, days, weeks, or months later, whenever you are feeling in the mood to hear differences using your ears, not eyes.
He ends by mentioning he likes "blind" A/B testing, switching back and forth quickly between A and B, (with a helper in the room). He's done a splendid job in this video of fooling people into thinking this exact same technique of quickly switching from A to B can not occur in ABX testing, whereas in truth many users doing ABX choose to do exactly this. I use this method quite often.
