Validty of blind testing - Page 2 - AVS Forum
Forum Jump: 
Closed Thread
 
Thread Tools
post #31 of 355 Old 02-03-2014, 01:26 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
To me, the important consideration in sighted vs. blind or double blind tests is that toole, olive and others have shown that sighted tests reveal bias that affects results. Yet, many audiophiles are convinced that their data is wrong and the tests are invalid, or they are not subject to sight bias.

Since I believe strongly in sight bias (as do most advertisers), i actually would place more stock in a dbt test even if it wasn't perfectly developed than I would an anecdotal subjective review. Just my opinion.
Randy Bessinger is offline  
Sponsored Links
Advertisement
 
post #32 of 355 Old 02-03-2014, 01:37 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 762 Post(s)
Liked: 1175
Quote:
Originally Posted by jim19611961 View Post

Quote:
Originally Posted by Ethan Winer View Post

Yes, exactly. I also agree that some type of fidelity loss need training to notice. But that's different from blind testing. If you haven't learned what to listen for, you won't spot it sighted either.

--Ethan

For me, this muddies the results of blind A/B testing because we would have to know how trained the listeners are. In one test, you could have a room full of untrained listeners that would skew the results in the direction of there being no differences (or smaller ones) between A & B.

Please do not make the mistake of of demonizing blind A/B testing because this requirement for trained listeners is not unique to them. Ethan said so in the quoted text and here we go again, only blind tests are criticized. Any subjective test has a certain amount of mud in it!
Quote:
And for all practical purposes, I know of no way to evaluate or test listeners to determine how acute their listening discernment really is

It is very easy to evaluate listeners to determine how acute their listening discernment is. You simply provide them with a series of stimuli with the artifact in question at decreasing levels, mixed with samples with no artifacts and determine when they cease to be able to tell the difference by just listening.
Quote:
And even if such a test exists, the blind A/B tests dont share this kind of info about the listeners in these tests.

There is actually quite a bit of sharing of notes by different practitioners of the blind A/B testing art. It shows up in professional journals and on specialist forums that some may not be privy to.
Quote:
So its possible that a majority of blind A/B results show that people cant tell a difference not because there isn't one, but because they are not trained to notice the differences.

The issue of listener training comes up again and again in the literature of blind A/B testing. ITU recommendation BS1116-1 . In addition ITU-R BS.1534-1 goes over the same basic information in the context of slightly different testing strategy that is oriented towards perceptual encoders that make no pretense about sonic transparency. Google is your friend!

Also: https://tech.ebu.ch/docs/tech/tech3324.pdf "EBU Evaluations of Multichannel Audio Codecs"
arnyk is offline  
post #33 of 355 Old 02-03-2014, 01:40 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 762 Post(s)
Liked: 1175
Quote:
Originally Posted by arnyk View Post

Quote:
Originally Posted by nvidio View Post

During DBT, do the test subjects focus their hearing mainly on those specific characteristics of the sound that will give them the best chance of real differences being revealed to them, or do they focus their hearing mainly on those specific characteristics of the sound that they expect to give them the best chance of real differences being revealed to them? biggrin.gif

Part of listener training is to make the two goals be one and the same.

Concering a statement above:

It is very easy to evaluate listeners to determine how acute their listening discernment is. You simply provide them with a series of stimuli with the artifact in question at decreasing levels, mixed with samples with no artifacts and determine when they cease to be able to tell the difference by just listening.

The series of stimuli starts out with the artifact at such a high level that it is practically unmistakable.
arnyk is offline  
post #34 of 355 Old 02-03-2014, 01:45 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by Randy Bessinger View Post

To me, the important consideration in sighted vs. blind or double blind tests is that toole, olive and others have shown that sighted tests reveal bias that affects results. Yet, many audiophiles are convinced that their data is wrong and the tests are invalid, or they are not subject to sight bias.

Since I believe strongly in sight bias (as do most advertisers), i actually would place more stock in a dbt test even if it wasn't perfectly developed than I would an anecdotal subjective review. Just my opinion.
I can show you an example of a DBT that proves beyond the shadow of a doubt the fact there is no valid reason to assume that it is even technically possible to design a valid listening test of any kind, let alone conduct one.
nvidio is offline  
post #35 of 355 Old 02-03-2014, 01:49 PM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 4,036
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 117 Post(s)
Liked: 165
Quote:
Originally Posted by nvidio View Post

I can show you an example of a DBT that proves beyond the shadow of a doubt the fact there is no valid reason to assume that it is even technically possible to design a valid listening test of any kind, let alone conduct one.

whip it out.
JHAz is offline  
post #36 of 355 Old 02-03-2014, 01:51 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,185
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 123 Post(s)
Liked: 324
Quote:
In one test, you could have a room full of untrained listeners that would skew the results in the direction of there being no differences (or smaller ones) between A & B.
Indeed you could, in one test. But this cannot be true for all tests. So the question you need to address is, why have there been no documented DBTs showing that humans can distinguish between conventional DACs or amps operating below clipping? Surely the industry and its media cheerleaders have a profound economic interest in proving the settled science wrong. Why have they failed to do so?

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is online now  
post #37 of 355 Old 02-03-2014, 01:54 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,185
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 123 Post(s)
Liked: 324
Quote:
ITU-R BS.1116-1 not only requires the use of expert listeners,

3.1 Expert listeners
It is important that data from listening tests assessing small impairments in audio systems should come exclusively from subjects who have expertise in detecting these small impairments. The higher the quality reached by the systems to be tested, the more important it is to have expert listeners. etc.

but also stipulates pre and post screening, as well as guidelines for panel size. And, to circle back to an earlier point, it also stipulates the use of suitable "critical material" for the test(s). AFAICT and for better or worse, few tests I've seen mentioned rise to the level of rigor spelled out in the ITU guidance.
But the whack-job audiophile set claims it's already trained itself to hear these differences. So why can't they?

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is online now  
post #38 of 355 Old 02-03-2014, 02:01 PM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,406
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 134 Post(s)
Liked: 88
Quote:
Originally Posted by arnyk View Post

Please do not make the mistake of of demonizing blind A/B testing because this requirement for trained listeners is not unique to them. Ethan said so in the quoted text and here we go again, only blind tests are criticized. Any subjective test has a certain amount of mud in it!

It is very easy to evaluate listeners to determine how acute their listening discernment is. You simply provide them with a series of stimuli with the artifact in question at decreasing levels, mixed with samples with no artifacts and determine when they cease to be able to tell the difference by just listening.

There is actually quite a bit of sharing of notes by different practitioners of the blind A/B testing art. It shows up in professional journals and on specialist forums that some may not be privy to.

The issue of listener training comes up again and again in the literature of blind A/B testing. ITU recommendation BS1116-1 . In addition ITU-R BS.1534-1 goes over the same basic information in the context of slightly different testing strategy that is oriented towards perceptual encoders that make no pretense about sonic transparency. Google is your friend!

Also: https://tech.ebu.ch/docs/tech/tech3324.pdf "EBU Evaluations of Multichannel Audio Codecs"

The only thing I am attending to demonize is that DBT's in of themselves are subject to the specific issue I am laying out. No more, no less.

What I meant is that there is no way to compare the listening prowess of one group against another in a statistical and/or measured way. Sure, you could separate the trained from the untrained. But even amongst the trained, certainly there are variations that turn up under the test criteria you outline.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #39 of 355 Old 02-03-2014, 02:06 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by JHAz View Post

whip it out.
Just watch from 8:09 to 9:22.
nvidio is offline  
post #40 of 355 Old 02-03-2014, 02:13 PM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,406
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 134 Post(s)
Liked: 88
Quote:
Originally Posted by mcnarus View Post

Indeed you could, in one test. But this cannot be true for all tests. So the question you need to address is, why have there been no documented DBTs showing that humans can distinguish between conventional DACs or amps operating below clipping? Surely the industry and its media cheerleaders have a profound economic interest in proving the settled science wrong. Why have they failed to do so?

The vast majority of people changing/upgrading their system bring home their new gear and hear differences. If the opposite was the case (no difference), they would return their purchase. As scientifically flawed as this may be, I don't find it odd at all that most people don't apply scientific scrutiny to something they are convinced they can discern without it.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #41 of 355 Old 02-03-2014, 02:13 PM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 4,036
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 117 Post(s)
Liked: 165
I remember posting a long time ago something I ran across by following links from the Michael J Human amp thread showing that folks in some Scandinavian country, when tested blind, did not hear a carrier tone that was actually quite audible in one particular digital radio process. A lot of stuff hits our ears. What we notice is entirely different. But a trained listener knowing what to listen for can weed out the bad things. Good thing because once you finally tumbled to the carrier tone, I'm guessing you could never un-hear it under any circumstances.

Now, I'm not trying t say anybody's personal experience is a result of any deficiency. I'm a musician, I think I listen pretty closely. I think I hear pitch problems, for example, about as acutely as people are shown to be able to hear them. But there is absolutely zero question that the frequency response of any speaker with non-colocated tweeter and mid/woofer changes as you move off axis. Hell, every Stereophile test of any such speaker clearly shows the phenomenon. But I have never ever heard it, either casually or when explicitly listening for it. I suspect that it's just not what my brain is "trained" to listen for. I'm pretty sure I'm not much stupider than any other national merit scholar, and I feel pretty sure that I listen pretty acutely. But it's a real audible phenomenon that I simply never notice. The fact that I don't hear it does't make me either dumb or a bad person. But a giant stack of people like me would miss some sonic anomalies that others would find immediately noticeable. And probably would hear some that those other listeners miss. But if you've got both a wide range of "regular folks" plus listeners trained to hear specific anomalies, you should get at least reasonably near to the truth. The existence and effect of subconscious bias, and the utter inability of any actual human being to control their subconscious, seem fairly well established. We're left with blind testing as the gold standard like it or not.

Now, no properly set up and understood such test can prove beyond a shadow of a doubt that there is no difference. Set up in a way that you can use statistics to analyze the results, you can either prove or disprove a null hypothesis which has to be, "these two things sound the same." That's just how this particular scientific thang works. You can have a stack as high as Mt. Everest of results that do not disprove the null hypothesis, but you still haven't absolutely positively proven that there's not a difference. Just that a difference is unlikely to exist.

But if there are golden eared persons who reliably hear differences that the rest of us miss, that can be teased out through statistics, too, at least if you have a big enough testing population. I am not aware of anybody developing science to demonstrate the existence of such a group. And there are plenty of businesses with a vested interest in developing such proof if it were possible to do. I suspect the paucity of science proving the existence of golden ears is the result of a lot of failed efforts to do so that simply were not published because the sponsors didn't want them published.
JHAz is offline  
post #42 of 355 Old 02-03-2014, 02:26 PM
AVS Special Member
 
beaveav's Avatar
 
Join Date: Mar 2010
Location: SoCal
Posts: 1,906
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 92 Post(s)
Liked: 324
Quote:
Originally Posted by R Swerdlow View Post

Good point.

In the medical and pharmaceutical world, before anyone can publish clinical trial results, they have to clear a much higher bar than in the audio industry. They have to positively document efficacy, not just rely on hearsay. For example, in cancer drugs, they have to shrink tumors, and provide statistically significant longer patient survival times. The Food and Drug Administration acts as another level of referee; they often decide just how high or low the efficacy bar should be set. In addition, they have legal regulatory authority over all clinical trials of experimental drugs. Its very expensive, and more often than not, a trial has negative results.

Nothing like that exists in audio. I can easily imagine people in the clinical trial business wishing they had it as easy and people in audio biggrin.gif.

Good post, and true, but I want to point out that the "bar" isn't always an easily measured parametric such as tumor size. Take the case of anti-depressants, for example. There is no scan, or blood test, or urinalysis, that will tell whether an anti-depressant is working. The only measures are questionnaires asking the patient how they are feeling. Now, these questionnaires are carefully crafted, and there is much debate as to how to optimize those questions.

And there are other, non-psychiatric conditions where the efficacy, or lack thereof, of a potential treatment can't be measured with any simple parameter, and drug trials have to rely on a combination of clinical evidence (scans, blood markers, etc) along with questionnaires.

For every new thing I learn, I forget two things I used to know.
beaveav is offline  
post #43 of 355 Old 02-03-2014, 02:26 PM
AVS Special Member
 
BruceD's Avatar
 
Join Date: Oct 2000
Location: Silicon Valley, CA USA
Posts: 1,064
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 29
Quote:
Originally Posted by jim19611961 View Post

The vast majority of people changing/upgrading their system bring home their new gear and hear differences. . . . .

Pure bias-induced satisfaction after spending money, what do you expect ?

You think people are inclined to distrust their own selections that cost real dollars ?

This is exactly what is being discussed here.

If a person is happy with his selection and it's characteristics, great, just don't proclaim any predictive results from that personal decision, it doesn't apply.
BruceD is offline  
post #44 of 355 Old 02-03-2014, 02:27 PM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 4,036
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 117 Post(s)
Liked: 165
Quote:
Originally Posted by nvidio View Post

Just watch from 8:09 to 9:22.

so not about double blind testing at all, just about how we hear what we expect. Expectation bias is what double blind testing attempts to remove from the testing process. It may not be perfect, but it's the best we have.

Otherwise, let's just put everything on the market as curing everything, because somebody will think they're healed of cancer by a sugar pill . . .
JHAz is offline  
post #45 of 355 Old 02-03-2014, 02:31 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
Quote:
Originally Posted by nvidio View Post

Quote:
Originally Posted by JHAz View Post

whip it out.
Just watch from 8:09 to 9:22.
The first one seemed an example of sight bias and not a DBT to me. The second was a case of not listening closely. I did not hear an "s" in legislators for what it is worth. Interesting but really not anything new. I am an old guy and remember the Kingsmen's Louie Louie song. If you are old you will know what I am talking about. Where is Ethan when you need him as I am not sure he would agree those examples completely eliminated any possible DBT conclusions in audio. I am sure Toole and Olive would also disagree with you.
Randy Bessinger is offline  
post #46 of 355 Old 02-03-2014, 02:32 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Quote:
Originally Posted by Jack D Ripper View Post

That a test is blind does not guarantee that it is a good one.  The reason being, of course, that there are many things necessary for a test to be good. 
Right. And how do you all those necessary things??? From what I have seen, some folks accept the results of any double blind tests that fits their view of audio, and ask questions later smile.gif. It is actually worse than that. Even after someone like me points out the problems in a test, folks continue to believe. Moreover, they get emotional like someone took away their first child!

What does it take for you to accept a test is not good? How much due diligence do you do to know the test is good? How much do you know about the systems under test to know the right conditions were created for it to be revealing and hence good?
Quote:
But if it is not blind, then it isn't a good one.  The reason being, of course, that otherwise bias can contaminate the results.
If we stuck to this rule, we would not have advanced much beyond Edison's phonograph. And our kids would be angry at us because they would have to haul this to school to listen to music:

230px-EdisonPhonograph.jpg

99% of how audio is advanced is done without blind testing. The key is not whether the test is blind. The key is whether the results are trustworthy. For example almost all research into acoustics and our hearing system is done sighted. See this short set of posts where I point out to Arny the same thing: http://www.avsforum.com/t/1492007/external-dac-questions/240#post_24289642. Surprisingly he thinks those are double blind tests!

In the industry, we use sighted evaluation by experts for almost all of the development. We then occasionally perform double blind tests to double check our work. Double blind tests are expensive and very slow. If you have a researcher working 8 hours a day tweaking say, the psychoacoustics of a codec encoder, it simply is not feasible to have him make a one line code change and then wait a week for someone to run double blind tests. They check it with their own ear and a trusted expert listener or two. I performed that role for my team while at Microsoft. I could give them quick feedback as to whether we moved forward or backward. Since my career depended on being right, you can bet we would not fool ourselves by believing in placebo as opposed to real differences. If we did, competitors would surely make us look bad once we released the system. Today, billions (yes, billions) of devices run technology created from that group from every Android phone to every blu-ray player. The latter came specifically as a result of double blind tests that showed our technology to be superior. Yet 99% of the development of that came from non-blind tests.

You tell me how you can make a living advancing audio technology if the only data you trusted was double blind. And why you think the expert listeners must be wrong.
Quote:
In other words, being blind is one of the necessary features of a good test, but it is not the only necessary feature of a good test.  That one can come up with bad tests that are done blind does nothing to show that there is no need for testing to be blind.
No need for blind tests? Who said that? Of course we need blind tests. It is just that we don't ask for a blind test to see if it is OK to breath smile.gif.

The problem is that people not only blindly believe such tests, they extrapolate to every other product and listener in the future. Another personal example smile.gif. I wanted to test the difference between a blu-ray player decoding an audio stream and an AVR. I bought two identical Onkyo AVRs and two identical Blu-ray players. That would then allow me to do instant switching of the output of the two AVRs (through their pre-amp out to my headphones). Being the careful person I am, I put the AVRs on my test bench and measured them. To my surprise, one of them had a high frequency oscillation in it in the audible band! It had much elevated noise floor compared to the first sample. Now how many people do you know that have a $20,000 test instrument to test the assumption that two "identical" AVRs have the same performance? I got disgusted and gave up on testing. I had already wasted a whole weekend on it setting up the blind test and of course lots of money. The answer was not worth more than that to me smile.gif.

If I want to buy a Yamaha or Pioneer AVR today, what double blind test data do you have for me to look at? Do you have any double blind tests of AVRs? How about DACs? Any tests of current products in the market? Or even last year's model? How about CD players?

Then there is the problem of believing what is publicized versus all that there is. You saw Arny quote the Meyer and Moran test of high resolution audio. But how often do you see this published AES report: https://secure.aes.org/forum/pubs/conventions/?elib=12799

I suspect you have not ever heard of it. Here is a bit of explanation on it: http://www.whatsbestforum.com/showthread.php?2316-How-The-Ear-Works&p=29514&viewfull=1#post29514

From that post and text in the double blind study:

"The four highest scores fell into the region of “critical probability.” This amounted to only 2.76% of all the tests. These four tests were carried out by four separate listeners, all of whom chose stereo music examples, and in all four cases headphones were used—thus excluding the influence of the listening environment to the greatest possible extent."

So four people probably heard the difference between SACD and DVD-Audio! Did you know about this test?

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is online now  
post #47 of 355 Old 02-03-2014, 02:32 PM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,406
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 134 Post(s)
Liked: 88
Quote:
Originally Posted by BruceD View Post

Pure bias-induced satisfaction after spending money, what do you expect ?

You think people are inclined to distrust their own selections that cost real dollars ?

This is exactly what is being discussed here.

If a person is happy with his selection and it's characteristics, great, just don't proclaim any predictive results from that personal decision, it doesn't apply.

I have made many a purchase or borrowed equipment that I expected to be an improvement that after listening, thought was worse than what I already had.

I have been treating my listening room for over a year. I have made dozens of changes with the hope every time that I will hear an improvement. Sometimes I do, sometimes not, sometimes cant tell the difference. So I don't weigh a predisposed bias as undermining objective audible discernment.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #48 of 355 Old 02-03-2014, 02:37 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by JHAz View Post

so not about double blind testing at all, just about how we hear what we expect. Expectation bias is what double blind testing attempts to remove from the testing process. It may not be perfect, but it's the best we have.

Otherwise, let's just put everything on the market as curing everything, because somebody will think they're healed of cancer by a sugar pill . . .
So, basically, you just admitted that the best we have is a test during which we hear sounds that aren't real. Meaning, the best we have are tests that are invalid. wink.gif
nvidio is offline  
post #49 of 355 Old 02-03-2014, 02:41 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
But don't you have to look at the flaws of a test and determine whether it means the test is worthless or has some value? Amir seems to at least imply that if a DBT has flaws, it is invalid or means that the opposite is a fact. So if someone can't distinguish between say two amps and Amir points out the flaws of the test, you cannot automatically conclude that there were differences. That is my beef when I look at the overwhelming evidence of being fooled by both sight and sound, to suddenly start defending the golden ear audiophiles. It is just too much of a stretch for me.
Randy Bessinger is offline  
post #50 of 355 Old 02-03-2014, 02:49 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by Jack D Ripper View Post

That a test is blind does not guarantee that it is a good one.  The reason being, of course, that there are many things necessary for a test to be good. 
Right. And how do you all those necessary things??? From what I have seen, some folks accept the results of any double blind tests that fits their view of audio, and ask questions later smile.gif. It is actually worse than that. Even after someone like me points out the problems in a test, folks continue to believe. Moreover, they get emotional like someone took away their first child!

What does it take for you to accept a test is not good? How much due diligence do you do to know the test is good? How much do you know about the systems under test to know the right conditions were created for it to be revealing and hence good?
Quote:
But if it is not blind, then it isn't a good one.  The reason being, of course, that otherwise bias can contaminate the results.
If we stuck to this rule, we would not have advanced much beyond Edison's phonograph. And our kids would be angry at us because they would have to haul this to school to listen to music:

230px-EdisonPhonograph.jpg

So four people probably heard the difference between SACD and DVD-Audio! Did you know about this test?
Probably???
Randy Bessinger is offline  
post #51 of 355 Old 02-03-2014, 03:03 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Quote:
Originally Posted by Randy Bessinger View Post

Probably???
What did you expect instead???

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is online now  
post #52 of 355 Old 02-03-2014, 03:08 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by Randy Bessinger View Post

Probably???
What did you expect instead???
I am not sure, that is why I had question marks. Do you think they could distinguish or not? By probably, does that indicate your agreement or are you hedging your bets:-)

I have not looked at the test so I assumed the probably statement at the end was yours. Is that not the case? Did the scientific study conclude they "probably" heard a difference. In accounting and law, there are opinions that are pretty standardized such as "more likely than not" etc. that equal percentages. I assume that probably in this case would be at least 51% or perhaps more...I am asking not telling.
Randy Bessinger is offline  
post #53 of 355 Old 02-03-2014, 03:16 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,185
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 123 Post(s)
Liked: 324
Quote:
The vast majority of people changing/upgrading their system bring home their new gear and imagine they hear differences.
Fixed.
Quote:
As scientifically flawed as this may be, I don't find it odd at all that most people don't apply scientific scrutiny to something they are convinced they can discern without it.
Nor do I. Most people are intellectually incurious enough to believe their ears. But the fact that these differences generally disappear when scientific scrutiny is applied should tell you something.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is online now  
post #54 of 355 Old 02-03-2014, 03:18 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by Randy Bessinger View Post

The first one seemed an example of sight bias and not a DBT to me.
Read my post again, please. I was referring to the one that starts 8:09 into the video.
Quote:
The second was a case of not listening closely. I did not hear an "s" in legislators for what it is worth. Interesting but really not anything new.
The vast majority of people who understand spoken English actually reported that they did hear an "s". This is because people are programmed to hear speech in such way that they hear sounds that aren't real, as was carefully explained in the video. Is there any reliable objective evidence to support the claim that people aren't also programmed to hear sounds that are unrelated to speech, and that also aren't real? Even if we can reliably assume that there is such reliable objective evidence, doesn't the experiment shown in the video reliably objectively prove that, if a DBT uses test samples that contain speech, this DBT is necessarily going to be invalid? And, if we can reliably assume that the experiment does objectively reliably prove this, does it mean that I should quit enjoying recorded music that contains speech because then there is no way for me to reliably tell how well or how bad my stereo playback system is capable of reproducing recorded speech? biggrin.gif
nvidio is offline  
post #55 of 355 Old 02-03-2014, 03:24 PM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,342
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 65 Post(s)
Liked: 67
Quote:
Originally Posted by nvidio View Post

Quote:
Originally Posted by Randy Bessinger View Post

The first one seemed an example of sight bias and not a DBT to me.
Read my post again, please. I was referring to the one that starts 8:09 into the video.
Quote:
The second was a case of not listening closely. I did not hear an "s" in legislators for what it is worth. Interesting but really not anything new.
The vast majority of people who understand spoken English actually reported that they did hear an "s". This is because people are programmed to hear speech in such way that they hear sounds that aren't real, as was carefully explained in the video. Is there any reliable objective evidence to support the claim that people aren't also programmed to hear sounds that are unrelated to speech, and that also aren't real? Even if we can reliably assume that there is such reliable objective evidence, doesn't the experiment shown in the video reliably objectively prove that, if a DBT uses test samples that contain speech, this DBT is necessarily going to be invalid? And, if we can reliably assume that the experiment does objectively reliably prove this, does it mean that I should quit enjoying recorded music that contains speech because then there is no way for me to reliably tell how well or how bad my stereo playback system is capable of reproducing recorded speech? biggrin.gif
Yes, you should quit listening and enjoying music.biggrin.gif
Randy Bessinger is offline  
post #56 of 355 Old 02-03-2014, 03:26 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by Randy Bessinger View Post

Yes, you should quit listening and enjoying music.biggrin.gif
tongue.gif
nvidio is offline  
post #57 of 355 Old 02-03-2014, 03:51 PM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 4,036
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 117 Post(s)
Liked: 165
Quote:
Originally Posted by nvidio View Post

So, basically, you just admitted that the best we have is a test during which we hear sounds that aren't real. Meaning, the best we have are tests that are invalid. wink.gif

Well, since we are all human beings, we have to deal with the realities of human perception. I'm not suggesting people don't hear what they hear, or not hear what they don't hear. Just like I'd never suggest that the person whose headache went away after a sugar pill, or whose cancer went into remission when they took a placebo didn't really lose their headache or go into remission. The question is whether the response is because of the placebo. Uhhh, no.

As long as you are human, you, too are subject to all the frailties that humanit entails, and we do the best we can to control for those in scientific endeavors. But maybe a person could sell the dickens out of some elixer, door to door, if they're not bothered by those minor details of whether an anecdote is proof . . .
JHAz is offline  
post #58 of 355 Old 02-03-2014, 04:40 PM
Senior Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 261
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 78 Post(s)
Liked: 25
Quote:
Originally Posted by JHAz View Post

Well, since we are all human beings, we have to deal with the realities of human perception. I'm not suggesting people don't hear what they hear, or not hear what they don't hear. Just like I'd never suggest that the person whose headache went away after a sugar pill, or whose cancer went into remission when they took a placebo didn't really lose their headache or go into remission. The question is whether the response is because of the placebo. Uhhh, no.

As long as you are human, you, too are subject to all the frailties that humanit entails, and we do the best we can to control for those in scientific endeavors. But maybe a person could sell the dickens out of some elixer, door to door, if they're not bothered by those minor details of whether an anecdote is proof . . .
Are you now trying to suggest the conclusions drawn from the blind listening test that is portrayed in the video are somehow unscientific? If yes, please explain why, and please also back up your explanation with something other than anecdotal evidence.

P.S. - The question is not "do we have to deal with the realities of human perception?"; the question is "can we?"...

P.P.S. -
Quote:
and we do the best we can to control for those in scientific endeavors
That appears to heavily depend on how exactly you choose to define "we". biggrin.gif
nvidio is offline  
post #59 of 355 Old 02-03-2014, 05:41 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,185
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 123 Post(s)
Liked: 324
Quote:
The question is not "do we have to deal with the realities of human perception?"; the question is "can we?"...
Of course we can. I mean, we've only been studying it for a century and a half or more. That you seem totally unfamiliar with this work does not mean the work has not been done.
Jack D Ripper likes this.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is online now  
post #60 of 355 Old 02-03-2014, 05:43 PM
AVS Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,360
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 177 Post(s)
Liked: 229
Quote:
Originally Posted by goneten View Post

Opponents of DBT testing will probably claim that the methodology is designed to result in negative outcomes. I think Robert Harley believes that DBTs are rigged and are therefore destined to result in false results. biggrin.gif

Since they do produce positive results (I've seen them for lossy codec settings, for loudspeakers, for different CD masterings, even for CD players and loudspeaker wire) , Robert Harley would appear to be wrong. Which is no surprise at all (but perhaps I'm biased).

Typically where they produce a 'positive', there is a routine measured difference of relatively large magnitude too. Funny how that works.
krabapple is offline  
Closed Thread Audio Theory, Setup, and Chat

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off