Validty of blind testing - Page 11 - AVS Forum

AVS Forum > Audio > Audio Theory, Setup, and Chat > Validty of blind testing

Audio Theory, Setup, and Chat

krabapple's Avatar krabapple
11:21 AM Liked: 229
post #301 of 355
02-06-2014 | Posts: 5,360
Joined: Apr 2003
Quote:
Originally Posted by jim19611961 View Post

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one? One that would be noticeable in a sighted test?

Because accurate 'difference' results can be readily achieved in a DBT? With virtually *anyone* who isn't stone deaf?

Let's start with a large magnitude difference, and test the subject double blind. Then gradually reduce the difference. You will absolutely find that the subject correctly tells A from B at some levels, and then performs less well as the objective difference decreases, finally reaching a point where they do no better than chance. Just where that threshold falls can vary from individual to individual based on their native acuity (either born that way, or trained) for that difference. You're saying that some mysterious DBT-related bias is what determines that threshold?
RobertR's Avatar RobertR
11:26 AM Liked: 423
post #302 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by krabapple View Post

The lack of qualifiers -- which you almost always see in scientific literature, e.g., 'our results make it likely that / our results suggest " that -- is one of the marks of a classic 'audiophile' claim. You rarely see acknowledgement that they *might* have heard wrong, *might* have been influenced by non-audio factors....
things which in science, are a *given*.
Yes, and it's fascinating to look at the vehemence of these unqualified "there is a NOTICEABLE, unmistakable difference" declarative statements (accompanied by paragraph after paragraph of extravagant prose detailing exactly how the difference manifests itself). In fact, such statements would lead one to believe that ANY listening test would be a cinch to pass with one hundred percent accuracy (no statistical analysis needed). But when confronted with the spectre of a DBT, suddenly, the "noticeable, unmistakable" difference becomes as unsubstantial as gossamer, vaporizing because there's "too much stress", or "not enough time to hear it", etc., and they only need to get this "unmistakable difference" mostly right.
krabapple's Avatar krabapple
11:29 AM Liked: 229
post #303 of 355
02-06-2014 | Posts: 5,360
Joined: Apr 2003
Quote:
Look let me sum it up this way. One of these statements I am onboard with. The other I am not.

A) People cannot discern audible differences using sighted tests. They are always wrong. DBT's are always right.

B) DBT's maybe more accurate than sighted tests.



A) is a strawman argument, i.e., one that NO ONE is making, and it's disingenuous of you to suggest otherwise.

B) is a truism with typos in it.
amirm's Avatar amirm
11:33 AM Liked: 547
post #304 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by arnyk View Post

I see none of the evidence that I asked for, and I do see more insults, such as the claim that the magazine article that I cited was "embarassing". I see no documentation of your claims. Therefore i have every reason to doubt the good faith of this reply. No deal.
I said your *answer* was embarrassing in that you provided only one example of an out of print article that was the work of three people and that was it. I didn't say the article was. Do you have a copy of the article so that we can read? And precisely what was your role as compared to the other two authors?

As for me, I provided a very detailed documentation of the watermark test a few posts above. Do you have anything like that which you have ever post online? I have searched through many of your posts in usenet forums and have not ran into a single one there either. I am starting to think of the shocking conclusion that maybe you have never shared such experiences.

Here is a call to everyone else who follows your posts in this forum. Have any of you seen him document any specific examples as I have been doing?
krabapple's Avatar krabapple
11:34 AM Liked: 229
post #305 of 355
02-06-2014 | Posts: 5,360
Joined: Apr 2003
Quote:
Originally Posted by RobertR View Post

Yes, and it's fascinating to look at the vehemence of these unqualified "there is a NOTICEABLE, unmistakable difference" declarative statements (accompanied by paragraph after paragraph of extravagant prose detailing exactly how the difference manifests itself). In fact, such statements would lead one to believe that ANY listening test would be a cinch to pass with one hundred percent accuracy (no statistical analysis needed). But when confronted with the spectre of a DBT, suddenly, the "noticeable, unmistakable" difference becomes as unsubstantial as gossamer, vaporizing because there's "too much stress", or "not enough time to hear it", etc., and they only need to get this "unmistakable difference" mostly right.

The corrollary being that, the DBT-bias effect jim is hypothesizing (as many DBT 'skeptics' have before him, in their more sophistical modes) must be in play even for *MASSIVE* difference like these ones audiophiles routinely report. Because these too have a distressing tendency to disappear under DBT conditions.

Such a powerful bias, yet so far unheralded by science -- and potentially devastating to at least some branches of it. Must be an AES conspiracy!
RobertR's Avatar RobertR
11:38 AM Liked: 423
post #306 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by amirm View Post


As for me, I provided a very detailed documentation of the watermark test a few posts above.
You gave essentially zero documentation of the protocols used, rendering your "detailed documentation" useless.
amirm's Avatar amirm
11:41 AM Liked: 547
post #307 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by RobertR View Post

You gave essentially zero documentation of the protocols used, rendering your "detailed documentation" useless.
Really? What else you wanted me to document? And do you have any examples like that which you have shared? That of Arny? Bruce or anyone else posting now? I like to read and evaluate how much you include in them versus mine.
RobertR's Avatar RobertR
11:41 AM Liked: 423
post #308 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by jim19611961 View Post

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one?
You seem to be suggesting that differences cannot be heard unless one has absolute conviction that they exist, and that all sources of doubt must be purged from one's consciousness.
RobertR's Avatar RobertR
11:43 AM Liked: 423
post #309 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by amirm View Post

Really? What else you wanted me to document?
Try reading the article Arny referred to to get a clue. I should have thought the answer to be obvious.
krabapple's Avatar krabapple
11:44 AM Liked: 229
post #310 of 355
02-06-2014 | Posts: 5,360
Joined: Apr 2003
I expect any page now we'll be down to debating whether cause and effect are real or an illusion.
arnyk's Avatar arnyk
11:44 AM Liked: 1178
post #311 of 355
02-06-2014 | Posts: 14,387
Joined: Oct 2002
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by RobertR View Post

You gave essentially zero documentation of the protocols used, rendering your "detailed documentation" useless.
Really? What else you wanted me to document? And do you have any examples like that which you have shared? That of Arny? Bruce or anyone else posting now? I like to read and evaluate how much you include in them versus mine.

Please provide us with documentation on an equal level with that which I have already provided such as an article in a well-known audio publication (HFN&RR), professional journal (JAES), on a web site (www.pcabx.com)., or even just in this thread.

Currrent score: Arny 4, Amir 0.
amirm's Avatar amirm
11:47 AM Liked: 547
post #312 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by RobertR View Post

Try reading the article Arny referred to to get a clue. I should have thought the answer to be obvious.
Oh, you have read the article? Beautiful! Can you please provide the summary and how you managed to get a copy? I am very interested in reading it.

I am curious about how well you document your tests online. Do you have any links? Any for Arny? Bruce, Krab, etc.? Remember I was not writing an article but sharing personal experience with sufficient detail for someone to understand what was done. I am very interested in like reports but seems like you guys don't have any. How about sharing one now if you never have before?
amirm's Avatar amirm
11:49 AM Liked: 547
post #313 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by arnyk View Post

Please provide us with documentation on an equal level with that which I have already provided such as an article in a well-known audio publication (HFN&RR), professional journal (JAES), on a web site (www.pcabx.com)., or even just in this thread.

Currrent score: Arny 4, Amir 0.
That link doesn't work Arny. I ask again: in the decades you have been online, have you ever explained any double blind tests you alone have created and ran? I ask for those since we can then read the thread. Giving me vague references with links that don't work doesn't allow us to learn from your experience.
RobertR's Avatar RobertR
11:53 AM Liked: 423
post #314 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by amirm View Post

I was not writing an article but sharing personal experience
IOW, you have no documentation, exactly as I thought.
jim19611961's Avatar jim19611961
12:02 PM Liked: 88
post #315 of 355
02-06-2014 | Posts: 1,421
Joined: Mar 2011
For what its worth, some of you have convinced me that doing a DBT sometime might be revealing.

Thanks.
amirm's Avatar amirm
12:04 PM Liked: 547
post #316 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by RobertR View Post

IOW, you have no documentation, exactly as I thought.
You mean other than explaining the full test and confirmed outcome??? wink.gifsmile.gif

I ask again Robert. Have you read the article that you said I need to follow since you are not forthcoming with what is missing in my explanation? How did you get a copy of the article?
amirm's Avatar amirm
12:05 PM Liked: 547
post #317 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by jim19611961 View Post

For what its worth, some of you have convinced me that doing a DBT sometime might be revealing.

Thanks.
It really is. These guys make the worst advocates for it in how they explain it. But done correctly it is essential to our understanding of audio in certain areas.
FMW's Avatar FMW
12:09 PM Liked: 831
post #318 of 355
02-06-2014 | Posts: 5,336
Joined: Jun 2007
Quote:
Originally Posted by jim19611961 View Post

1) No. The scientific method is sufficient. And that means the exact same conditions for each test with exactly the same outcome. NOT.

Says you.
Quote:
2) No. I am fine walking on a cement sidewalk without knowing the quantum state of each of its molecules biggrin.gif

Straw man
Quote:
3) Depends on the degree of the difference and what kind. Changing the temperature or humidity of the room makes some difference.

Says you.
Quote:
4) No. You dont get it.

I guess that settles it.
Quote:
5) Many believe Christ was resurrected. Is that a fact or belief?

Another straw man deflection.

Look let me sum it up this way. One of these statements I am onboard with. The other I am not.
Quote:
A) People cannot discern audible differences using sighted tests. They are always wrong. DBT's are always right

I'm not on board with that one. Are you?.
Quote:
B) DBT's maybe more accurate than sighted tests.

Well that is certainly a giant step forward. You are almost there.
RobertR's Avatar RobertR
12:10 PM Liked: 423
post #319 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by amirm View Post

You mean other than explaining the full test
You didn't. Zero detail about the protocols.
FMW's Avatar FMW
12:10 PM Liked: 831
post #320 of 355
02-06-2014 | Posts: 5,336
Joined: Jun 2007
Quote:
Originally Posted by jim19611961 View Post

For what its worth, some of you have convinced me that doing a DBT sometime might be revealing.

Thanks.

No reason to think otherwise. It certainly has been an ear opener for those of us who have done it.
RobertR's Avatar RobertR
12:11 PM Liked: 423
post #321 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by jim19611961 View Post

For what its worth, some of you have convinced me that doing a DBT sometime might be revealing.

Thanks.
Congratulations, Jim. Glad to see this discussion has been useful for you. smile.gif
amirm's Avatar amirm
12:14 PM Liked: 547
post #322 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by arnyk View Post

Ten (10) Requirements For Sensitive and Reliable Listening Tests

(1) Program material must include critical passages that enable audible differences to be most easily heard.
Wholeheartedly agree. So can you please repost your blind experiences with cables, amps, etc. and let's examine how this was achieved?
Quote:
(2) Listeners must be sensitized to a audible differences, so that if an audible difference is generated by the equipment, the listener will notice it and have a useful reaction to it.
Again, fully agree. Evidence of this would be very useful too smile.gif.
Quote:
(3) Listeners must be trained to listen systematically so that audible problems are heard.
Right on brother smile.gif.
Quote:
(4) Procedures should be "open" to detecting problems that aren't necessarily technically well-understood or even expected, at this time. A classic problem with measurements and some listening tests is that each one focuses on one or only a few problems, allowing others to escape notice.
Another good one.
Quote:
(5) We must have confidence that the Unit Under Test (UUT) is representative of the kind of equipment it represents. In other words the UUT must not be broken, it must not be appreciably modified in some secret way, and must not be the wrong make or model, among other things.
Goes without saying.
Quote:
(6) A suitable listening environment must be provided. It can't be too dull, too bright, too noisy, too reverberant, or too harsh. The speakers and other components have to be sufficiently free from distortion, the room must be noise-free, etc..
I like to use headphones for this reason if we are not dealing with speakers/room. They allow me to listen at very elevated levels. For equipment I get the most transparent and distortion free that I can get.
Quote:
(7) Listeners need to be in a good mood for listening, in good physical condition (no blocked-up ears!), and be well-trained for hearing deficiencies in the reproduced sound.
Yeh, I try to not do the listening test after being on AVS Forum in one of these threads. biggrin.gif Seriously, the heated arguments is what prompts me often to actually go and conduct such tests to get more data.
Quote:
(8) Sample volume levels need to be matched to each other or else the listeners will perceive differences that are simply due to volume differences.
You tell them Arny. smile.gif
Quote:
(9) Non-audible influences need to be controlled so that the listener reaches his conclusions due to "Just listening".
Yup!
Quote:
(10) Listeners should control as many of the aspects of the listening test as possible. Self-controlled tests usually facilitate this. Most importantly, they should be able to switch among the alternatives at times of their choosing. The switchover should be as instantaneous and non-disruptive as possible.
Absolutely essential. Testing becomes harder when someone else controls the switching.

I may have missed something but I think my watermark example that I gave complied with all of this. smile.gif
CharlesJ's Avatar CharlesJ
12:17 PM Liked: 117
post #323 of 355
02-06-2014 | Posts: 3,413
Joined: Jan 2006
Quote:
Originally Posted by krabapple View Post

You thought wrong.

Many an audiophile claims they are not significantly influenced by what they see, or know in advance. They consciously think they are free of such biases. They believe they are relying entirely on what their ears (or nose, or tongue, if we're talking wine tasters) tell them.

Years of sensory and product -choice testing data tell another tale on them.

It is not just consciously acknowledged or denied biases at work.

Btw, people typically claim to be nonracist and nonsexist, too. Very sincerely.

Yet when psychological testing is applied....we learn again and again that we aren't always consciously aware of our biases.
Jim gave the exact reasons why a DBT must be used to detect audible differences and at the same time he is arguing against the need for such testing. Not sure why he cannot see that. Perhaps he is afraid of the outcome.
Jack D Ripper's Avatar Jack D Ripper
12:23 PM Liked: 258
post #324 of 355
02-06-2014 | Posts: 1,136
Joined: Jan 2014
Quote:
Originally Posted by jim19611961 View Post
 
Quote:
Originally Posted by JHAz View Post

google placebo effect. google double blind testing. It's old science by now. Analytically, if we KNOW that knowledge of what one is testing skews the results, don't we have to know they're skewed versus the condition of not knowing? You have to start somewhere . . .

Your suggestion would conclude that blind testing of pharmaceuticals is also hopeless, so everybody should give up on medicines that are sold through the pharmacy (the hell with chemo) and buy stuff from the back of the wagon of the guy whose spiel you like the best. I'm guessing recovery rates and survival rates would plummet.

Purely anecdotally, i find that if I go to a concert where I like the sound but don't recognize the speakers, can't find the amp racks, and can't get close enough to see what board they're using FOH, I still can hear The trick for me is to stop wondering (also "what amp is that guitarist playing through?") and just enjoy the performance.



Please stop trying to compare audio listening test procedures to pharmaceutical drugs. Apples and oranges.

 

On the contrary, the biasses that one is trying to avoid are the same sort of thing.  If there were no human biasses, then double blind testing would not be needed in either case.  The fact that there are human biasses is precisely the reason why double blind testing is needed in both cases.


CharlesJ's Avatar CharlesJ
12:25 PM Liked: 117
post #325 of 355
02-06-2014 | Posts: 3,413
Joined: Jan 2006
Quote:
Originally Posted by krabapple View Post

...
B) is a truism with typos in it.
But that implies that sighted testing is accurate to some degree, no? And DBT is just more accurate.
Jack D Ripper's Avatar Jack D Ripper
12:32 PM Liked: 258
post #326 of 355
02-06-2014 | Posts: 1,136
Joined: Jan 2014
Quote:
Originally Posted by amirm View Post
 
...

1. That you don't need any technical knowledge to determine the usefulness of a blind test. -- Jack

...

 

You have a bad habit of misrepresenting what people state, and then mocking the imaginary position that no one asserted.

 

What I stated was that sometimes one can know that a test is useless, without technical knowledge specific to the test.  I never stated that one can always know that a test is useless without technical knowledge, nor did I state that one can know that it is useful without technical knowledge.  

 

It is also interesting that you did not provide a link to the post in which my claims were made.  Were you afraid that someone could too easily see that you grossly misrepresented what I stated?

 

Really, you desperately need to read up on the fallacy known as "straw man."  You keep committing that fallacy over and over.


David James's Avatar David James
12:36 PM Liked: 72
post #327 of 355
02-06-2014 | Posts: 1,945
Joined: Dec 2000
Just so I'm clear. Is it really being suggested that when taking a DBT, our brain could subconsciously push us towards not hearing a difference?
amirm's Avatar amirm
12:43 PM Liked: 547
post #328 of 355
02-06-2014 | Posts: 18,375
Joined: Jan 2002
Quote:
Originally Posted by David James View Post

Just so I'm clear. Is it really being suggested that when taking a DBT, our brain could subconsciously push us towards not hearing a difference?
Absolutely.
RobertR's Avatar RobertR
12:45 PM Liked: 423
post #329 of 355
02-06-2014 | Posts: 6,244
Joined: Dec 2000
Quote:
Originally Posted by David James View Post

Just so I'm clear. Is it really being suggested that when taking a DBT, our brain could subconsciously push us towards not hearing a difference?
Suggested in post #277, yes. Backed up by any evidence, no.
David James's Avatar David James
12:56 PM Liked: 72
post #330 of 355
02-06-2014 | Posts: 1,945
Joined: Dec 2000
Quote:
Originally Posted by amirm View Post

Absolutely.

Quote:
Originally Posted by RobertR View Post

Suggested in post #277, yes. Backed up by any evidence, no.

Thanks. I guess I'm not surprised, that's a variety of the same technique used by masters of all types of belief related "woo".

So basically it's the same tactic used by anti-science folks everywhere. Rather then provide positive proof of their position, they attempt to shoot holes in the science which disputes their beliefs. Somehow they think by succeeding it some how means their belief is true. Of course even if they were to successfully refute the science, the burden of proof still lye's with them.

Closed Thread Audio Theory, Setup, and Chat

Subscribe to this Thread

Powered by vBadvanced CMPS v3.2.3