Interesting Interview on ABX Audio Testing - AVS Forum
Forum Jump: 
 31Likes
Closed Thread
 
Thread Tools
post #1 of 100 Old 08-31-2014, 09:20 AM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Lightbulb Interesting Interview on ABX Audio Testing

https://www.youtube.com/watch?v=p6kC5vck2JA#t=490

Last edited by Garidy; 08-31-2014 at 06:20 PM.
Garidy is offline  
Sponsored Links
Advertisement
 
post #2 of 100 Old 08-31-2014, 07:58 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Good listen....

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #3 of 100 Old 08-31-2014, 08:46 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by Garidy View Post
When you listen to these files, remember that "Blind Test" generally means "Single Blind Test" and that a single blind test is just a sighted evaluation with the some of the appearance of a DBT , but not the real thing.
arnyk is offline  
post #4 of 100 Old 08-31-2014, 08:50 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by arnyk View Post
When you listen to these files, remember that "Blind Test" generally means "Single Blind Test" and that a single blind test is just a sighted evaluation with the some of the appearance of a DBT , but not the real thing.
I'm not sure I follow... I'll have to watch the interview again.
Cerumen likes this.
Garidy is offline  
post #5 of 100 Old 08-31-2014, 09:11 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by Garidy View Post
I'm not sure I follow... I'll have to watch the interview again.
It is true that DBTs and a SBTs are both blind tests, but that does not mean that they are any way equivalent even though they may look that way to a casual viewer of the test.

A DBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from both the tester(s), and the subject(s). There is nobody ho knows the correct identity of the unknowns that has any visibility at all to anybody who can possibly affect the outcome of the test.

A SBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from the subject(s) but not the tester(s) or other persons who are present during the test. There is now someone who knows the correct identity of the unknowns who has some kind of visibility to somebody who can possibly affect the outcome of the test.

For many years SBTs were considered to be adequate until the well-known case of "Clever Hans the Talking Horse" in the early 1800s. Clever Hans was a horse who amazingly enough tapped out correct answers to difficult questions with his hooves. The tests were SBTs in that someone who knew the correct answers was visible to the horse and communicated with the horse whether intentionally or unintentionally via his body language. When the tests were upgraded to DBTs by removing anybody who knew the correct answers from the presence of the horse, the horse suddenly lost his question-answering abilities.

A listening test can't be done with the listener acting totally in a vacuum. The listener needs to know which trial is the current trial, for example. The easiest way to do a double blind test is to prepare a script of the trials and correct identities of the unknowns and give it to someone who controls the technical side of the test, and for example does the switching. This person is completely concealed from everybody else who is involved. A second person calls out the trial numbers, which the concealed person uses to do the switching, and the listeners use to record their results.

The first innovation of ABX was to build a machine that was a mechanical test coordinator that controlled the unknowns and kept the listeners updated as to trial numbers, etc. The second innovation was the ABX comparison method itself which allows sighted evaluations for the purpose of learning what to listen for, and blind tests for the purpose of determining the outcome of the test.
arnyk is offline  
post #6 of 100 Old 08-31-2014, 10:52 PM
Senior Member
 
spkr's Avatar
 
Join Date: Apr 2013
Posts: 384
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 171 Post(s)
Liked: 94
In that video, Jon just plays bunch of word games. He says many things already posted by stereoeditor (John Atkinson, Stereophile Editor) on this forum. Yup, they have their business tactics memorized.
spkr is offline  
post #7 of 100 Old 09-01-2014, 06:21 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,336
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 475 Post(s)
Liked: 831
I think an single blind test can work fine as long as the tester knows what he is doing and is not visible by the listener. That way the only possible cues would be spoken and those are pretty easy to conceal. We did single blind tests and they resolved to the same results as ABX tests.
amirm likes this.
FMW is offline  
post #8 of 100 Old 09-01-2014, 06:37 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by FMW View Post
I think an single blind test can work fine as long as the tester knows what he is doing and is not visible by the listener. That way the only possible cues would be spoken and those are pretty easy to conceal. We did single blind tests and they resolved to the same results as ABX tests.
I can interpret the above as saying that there isn't a black and white difference between SBTs and DBTs and in fact there is a judgment call that is involved. Then I agree. The closer the SBT comes to being a DBT, the better.

If the person running the experiment is truly not visible to the listeners the listenerers may still be able to pick up cues based on his voice, but they don't necessarily have to. YMMV.

The difference between a SBT and a DBT can be as simple as having two people running tests as I described above, instead of just one.
arnyk is offline  
post #9 of 100 Old 09-01-2014, 12:08 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by arnyk View Post
It is true that DBTs and a SBTs are both blind tests, but that does not mean that they are any way equivalent even though they may look that way to a casual viewer of the test.

A DBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from both the tester(s), and the subject(s). There is nobody ho knows the correct identity of the unknowns that has any visibility at all to anybody who can possibly affect the outcome of the test.

A SBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from the subject(s) but not the tester(s) or other persons who are present during the test. There is now someone who knows the correct identity of the unknowns who has some kind of visibility to somebody who can possibly affect the outcome of the test.

For many years SBTs were considered to be adequate until the well-known case of "Clever Hans the Talking Horse" in the early 1800s. Clever Hans was a horse who amazingly enough tapped out correct answers to difficult questions with his hooves. The tests were SBTs in that someone who knew the correct answers was visible to the horse and communicated with the horse whether intentionally or unintentionally via his body language. When the tests were upgraded to DBTs by removing anybody who knew the correct answers from the presence of the horse, the horse suddenly lost his question-answering abilities.

A listening test can't be done with the listener acting totally in a vacuum. The listener needs to know which trial is the current trial, for example. The easiest way to do a double blind test is to prepare a script of the trials and correct identities of the unknowns and give it to someone who controls the technical side of the test, and for example does the switching. This person is completely concealed from everybody else who is involved. A second person calls out the trial numbers, which the concealed person uses to do the switching, and the listeners use to record their results.

The first innovation of ABX was to build a machine that was a mechanical test coordinator that controlled the unknowns and kept the listeners updated as to trial numbers, etc. The second innovation was the ABX comparison method itself which allows sighted evaluations for the purpose of learning what to listen for, and blind tests for the purpose of determining the outcome of the test.
Thank you for the additional clarification of your position and opinions with regards to both SBT's and DBT's.

As stated above, I will re-watch the interview and carefully weight what you have stated against the Q&A that has occurred between Scott and Jon.

Thanks again for your reply
Garidy is offline  
post #10 of 100 Old 09-01-2014, 12:10 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by spkr View Post
In that video, Jon just plays bunch of word games. He says many things already posted by stereoeditor (John Atkinson, Stereophile Editor) on this forum. Yup, they have their business tactics memorized.
Hey spkr:

I did note such myself!

It seems like there isn't much new, if anything, under the Audio Sun!
Garidy is offline  
post #11 of 100 Old 09-01-2014, 12:47 PM
AVS Special Member
 
RayDunzl's Avatar
 
Join Date: Sep 2013
Location: Tampa, Florida
Posts: 1,402
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 480 Post(s)
Liked: 679
Iverson's room


I'll be back later...


links::: 1.5RQ > digits > 1177a > OpenDRC-DI > DEQ2496 > DAC2 > KCT > FPB 350mcx > reQuest > Sweetspot
RayDunzl is online now  
post #12 of 100 Old 09-01-2014, 01:29 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by RayDunzl View Post
Iverson's room

An awesome view, in everyway!
Garidy is offline  
post #13 of 100 Old 09-01-2014, 02:44 PM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 4,043
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 119 Post(s)
Liked: 167
Quote:
Originally Posted by FMW View Post
I think an single blind test can work fine as long as the tester knows what he is doing and is not visible by the listener. That way the only possible cues would be spoken and those are pretty easy to conceal. We did single blind tests and they resolved to the same results as ABX tests.
I agree with you so much of the time but here we must depart. You need to read Blink. If it was easy to conceal the difference, double blind testing would not ever be needed,including in drug testing. But, because so much human communication is subconscious, if the fella giving the placebo knows its a placebo it quite literally effects, statistically, whether the fella getting the placebo gets better. Even with cancer, let alone with a headache. So, no, I don't think that audio testers have greater control over their subconscious communication than the humans that give drug tests. Sorry. Gotta be double blind to fully remove the biases that blind testing are intended to remove.
JHAz is offline  
post #14 of 100 Old 09-01-2014, 03:01 PM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,336
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 475 Post(s)
Liked: 831
Quote:
Originally Posted by JHAz View Post
I agree with you so much of the time but here we must depart. You need to read Blink. If it was easy to conceal the difference, double blind testing would not ever be needed,including in drug testing. But, because so much human communication is subconscious, if the fella giving the placebo knows its a placebo it quite literally effects, statistically, whether the fella getting the placebo gets better. Even with cancer, let alone with a headache. So, no, I don't think that audio testers have greater control over their subconscious communication than the humans that give drug tests. Sorry. Gotta be double blind to fully remove the biases that blind testing are intended to remove.
The only thing the tester said was "A or B?" The listener couldn't see him. Perfect? Perhaps not but it worked. We got the same results the ABX tests got.
FMW is offline  
post #15 of 100 Old 09-01-2014, 03:54 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by arnyk View Post
It is true that DBTs and a SBTs are both blind tests, but that does not mean that they are any way equivalent even though they may look that way to a casual viewer of the test.

A DBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from both the tester(s), and the subject(s). There is nobody ho knows the correct identity of the unknowns that has any visibility at all to anybody who can possibly affect the outcome of the test.

A SBT experiment is an experiment in which any information about the test that might lead to bias in the results is concealed from the subject(s) but not the tester(s) or other persons who are present during the test. There is now someone who knows the correct identity of the unknowns who has some kind of visibility to somebody who can possibly affect the outcome of the test.

For many years SBTs were considered to be adequate until the well-known case of "Clever Hans the Talking Horse" in the early 1800s. Clever Hans was a horse who amazingly enough tapped out correct answers to difficult questions with his hooves. The tests were SBTs in that someone who knew the correct answers was visible to the horse and communicated with the horse whether intentionally or unintentionally via his body language. When the tests were upgraded to DBTs by removing anybody who knew the correct answers from the presence of the horse, the horse suddenly lost his question-answering abilities.

A listening test can't be done with the listener acting totally in a vacuum. The listener needs to know which trial is the current trial, for example. The easiest way to do a double blind test is to prepare a script of the trials and correct identities of the unknowns and give it to someone who controls the technical side of the test, and for example does the switching. This person is completely concealed from everybody else who is involved. A second person calls out the trial numbers, which the concealed person uses to do the switching, and the listeners use to record their results.

The first innovation of ABX was to build a machine that was a mechanical test coordinator that controlled the unknowns and kept the listeners updated as to trial numbers, etc. The second innovation was the ABX comparison method itself which allows sighted evaluations for the purpose of learning what to listen for, and blind tests for the purpose of determining the outcome of the test.
So, I've had another viewing of the interview, and the thing that stuck out the most to me was Scott's laugh!

Outside of this, I found that the interview dealt minimally with DBT's or SBT's, but spoke more to the recorded source qualities and common challenges surrounding the development and retention of such, and the reproduction qualities of electronic devices themselves, more than the test methodology, which for me, was why I posted the link initially. - The conversations in these regards, are very informative, and worth revisiting from time-to-time. Jon's, analogies were easy to relate to and follow. I personally found him to be self effacing, and honest. He said nothing that bordered on heresy (I understand that you may perceive otherwise), and in fact, his statement predominately echoed, the majority beliefs of this forum.

With regards to SBT's and DBT's, Id say that while the title leads one to anticipate that the debate would heavily include a discuss about the goodness's and usefulness of such test, it frankly does not. What was discussed / shared was reasonable, non-combative, and in agreement with my personal sensitivities. I never felt at any point that I was in need of filtering out much, if any of what I was hearing, from entering my mainstream of thinking.

On the whole, the central theme of the interview is about porting a greater understanding of all of the variables, which contribute to one discernment of subjective audio qualities, not just under a SDT or DBT, but in general everyday enjoyment of ones system. For me, the weighting was put in the right places, far away from a focus on the qualities of SDT's and DBT's because frankly, we all have stereos, but less then 1% of us have access to such test, outside of format testing, via Foo Bar. But again, for me, anything outside of Red Book, is of insufficient fidelity, for my current level of proficiency as an audiophile. Hardware differences are what I'm personally paying the most attention to, these days, and if I was to endeavor down the path of the aforementioned test format, I'd personally use both variants, along side of my everyday listens, as a means to filter down to my final choices. But ultimately, as I have historically, I would simply use my home as the environment, and my ear-brain combination, to determine what I personal like the most, given the combination of options available to me. I have not, nor would I ever accept another's persons beliefs about the sonic qualities of any recording, or playback device; at best I would put the items mentioned on my list to personally audition, in my own home. If a retailer will not permit me to fully pay for the items, and take them home, with the understanding that I will be making many back and fourth exchanges, for a period of no less than 30-days, then they will not get my business. There is no other way, to determine, what's best for you - to you!

I haven't purchased new components in almost 10-years. The last time I did, I subjectively auditioned all of the components on my objective wish list, in every combination available; and I came to a final combination, after several weeks of auditions, in my home. My investment wasn't lofty, but it wasn't paltry either. I didn't utilize either of the controlled tests cited within this thread, I did however, use a basic A/B relay switch to audition each component. It wasn't a controlled setting outside of the levels always being match to at least .5dB. I always knew what I was listening to and never went 'blind' if you will. I have suffered no cognitive dissidence with regards to these purchases. I should also add that I spend very little time listing to the system in a critical manner anymore, as I am spending to much time in here (and with family of course)!

I am due for a DAC upgrade - for me, I am able to quickly discern sonic differences between DAC's, which makes it requisite for me to bring them home for audition. I agree with Jon, that the analog filters produce the most discernible, audible differences; as I also mentioned last month, in my thread summarizing the AVS debate on Jitter, between Amir vs. everyone else, if you will. At that time I stated that I would put my money on the filters, Jon's comments, in this regard, mirror my own. As a predominantly back ground music system, these days, I don't really need to upgrade my DAC, which is why I probably won't.


Personal auditioning, listening, enjoying of a system in ones home, as they intend to use it, day in, and day out, year after year, is the best test method of all. There is currently no universal test, subjective, objective or of hybrid, that can surpass the aforementioned.

I'm not sure that I paid attentions to your warnings or not...

Last edited by Garidy; 09-01-2014 at 04:05 PM.
Garidy is offline  
post #16 of 100 Old 09-01-2014, 04:02 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,190
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 125 Post(s)
Liked: 327
Quote:
The only thing the tester said was "A or B?" The listener couldn't see him. Perfect? Perhaps not but it worked. We got the same results the ABX tests got.
The thing to remember about listening tests is that there are only two possible outcomes:

1) Subject(s) reliably heard a difference.

2) Subject(s) failed to reliably hear a difference.

If you do a SBT and fail to reliably hear a difference, it can be safely assumed that you would have similarly failed to hear a difference in a DBT.

But if you do a SBT and get a statistically significant result, you cannot say that you reliably heard a difference. Maybe you did, or maybe the test administrator tipped you off, consciously or subconsciously.
arnyk, FreeFire and m. zillch like this.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is offline  
post #17 of 100 Old 09-01-2014, 04:03 PM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,190
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 125 Post(s)
Liked: 327
Quote:
I am due for a DAC upgrade - for me, I am able to quickly discern sonic differences between DAC's, which makes it requisite for me to bring them home for audition.
Sigh. Another hopeless case.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is offline  
post #18 of 100 Old 09-01-2014, 04:58 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by JHAz View Post
I agree with you so much of the time but here we must depart.

You need to read Blink.

If it was easy to conceal the difference, double blind testing would not ever be needed,including in drug testing. Would you not agree that drug testing, is the testing of the human immune system and critical organs, to accept and produce the intended outcomes from prescribed drug(s)?

In Audio DBT's there's an active participant, not a passive patient.

A patient cannot prevent a drug from helping them (you know the context of this statement, so please don't introduce absurdities, please). Nor can they do much to help improve a drugs effectiveness (outside of rest and a balanced diet). If it works in the majority of patients, it worked no debate! If it didn't no debate there either.

In audio DBT's, there are no similarities, outside of the desired blind elements, of the test formats. The participant has a major influence on the outcome/results and can be greatly influenced emotionally, as to sway their interpretation/understanding, or simply to suppress them from stating a conclusion that may be met with ridicule, or a negative labeling, etc., (there sure is a lot of that within AVS). These statements of course preclude a participants ability to judge such in the first place, which also makes audio blind tests decidedly different from blind drug test, again, the patients don't need any skills what-so-ever!

If you had an ear infection, and I gave you the appropriate antibiotic and you took it as prescribed, the infection would be eradicated, and you would be well again. No debate! (please leave out the absurdities relating to exceptions)

But, if I was to place you in a series of audio SDT's and DBT's, the only consistent outcome would be a null, and you have full control of these outcomes! Will what you want - all you want, you cannot make it happen.

But to be a valid test, hundreds of participants would need to be tested, and their results averaged, and the null would still stand, but now it would be an objectively obtained null. All series of audio SDT's and DBT's, that have been documented, of sufficient numbers in terms of participants, and tests, have decidedly produced nulls. There are several instances were individuals produce positive results consistently, but when their performances are averaged, amongst with the other samples, the null still occurs.

Blind Drug tests have produce millions of positive outcomes, blinded audio tests have produced only nulls, within large/sufficient samples, as to be potentially deemed, scientifically objective outcomes. To date, only conjecture exists. For every study that exists that might suggest / not prove otherwise, there's hundreds against it. Repeatability - and probability equal objective and subjective reality.

So what does this suggest? It suggest that blind test are useful and scientifically conclusive for use in developing facts about humans interactions with drugs, it also suggest that blind audio testing isn't effective at scientifically developing any conclusions about audible differences, amongst so-call linear electronic devices. A Null is a Null - at this juncture, the results are more telling of other things, outside of humans ability at large to perceive audible differences.


But, because so much human communication is subconscious, if the fella giving the placebo knows its a placebo it quite literally effects, statistically, whether the fella getting the placebo gets better. Even with cancer, let alone with a headache.

Placebo effect hasn't been verified scientifically or mathematically. It could be that something else is occurring, perhaps we've misnamed it, like we once misnamed the emptiness in space as ether! Also, placebo isn't an automatic, which is to say, if someone is told that is a sugar pill, that they still haven't recovered, nor has it stopped a real drug from being effective either, when the recipient was told that they only got sugar, but in fact they got the drug. What we do know, with repeatability, is that you cannot cured disease or general health issues, with placebo/sugar pills. They are closer to one offs then anything else, which are highly circumstantial. If it were otherwise, our pharmacy shelves would be filled with placebos, apposed to what they're currently filled with today - real drugs!

Lastly, if blind drug tests produced the same amount of nulls (zero conclusiveness) that blind audio tests did, the testing format would have to be abandoned for another, or lives would be lost; hmmm - something to think about!


So, no, I don't think that audio testers have greater control over their subconscious communication than the humans that give drug tests. Sorry. Gotta be double blind to fully remove the biases that blind testing are intended to remove. IMO bias cannot be removed completely, only mitigated. Checkout Ethan's Video a few postings below...Audio Myths Workshop.
reply in context above...

Last edited by Garidy; 09-01-2014 at 09:58 PM.
Garidy is offline  
post #19 of 100 Old 09-01-2014, 05:00 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Quote:
Originally Posted by mcnarus View Post
Sigh. Another hopeless case.
Unnecessary comment!

How would you feel if I referred to you in a derogatory manner?
Cerumen and acuity like this.
Garidy is offline  
post #20 of 100 Old 09-01-2014, 08:14 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Red face

Related: Audio Myths Workshop

Poppy sets us up for a quick fall...

https://www.youtube.com/watch?list=P...&v=BYTlN6wjcvQ
Garidy is offline  
post #21 of 100 Old 09-01-2014, 08:23 PM - Thread Starter
Senior Member
 
Join Date: Jul 2014
Location: Canada
Posts: 464
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 205 Post(s)
Liked: 69
Related: Chris Athens Video - Mastering

Chris is well known and respected for his craft.

Garidy is offline  
post #22 of 100 Old 09-01-2014, 08:39 PM
Senior Member
 
spkr's Avatar
 
Join Date: Apr 2013
Posts: 384
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 171 Post(s)
Liked: 94
Quote:
Originally Posted by Garidy View Post
and in fact, his statement predominately echoed, the majority beliefs of this forum.
When did you take the poll? I missed it.
spkr is offline  
post #23 of 100 Old 09-01-2014, 11:20 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by FMW View Post
The only thing the tester said was "A or B?" The listener couldn't see him. Perfect? Perhaps not but it worked. We got the same results the ABX tests got.
Unfortunately obtaining the same results as another test doesn't globally validate the first one.
arnyk is offline  
post #24 of 100 Old 09-01-2014, 11:26 PM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by mcnarus View Post
Sigh. Another hopeless case.
It is quite clear that the Stereophile line was swallowed hook, line, and sinker.

One of the tactics of propaganda is to surround the lies with a veneer of truth.

In this case the deception seems to have started right out with the title.

Even advocates for the video admit that other than praising the golden ears of a SP staffer based on a false conflation of SBTs and DBTs, as well as another false conflation of media mastering and equipment listening tests, the video wasn't even about blind testing.

Last edited by arnyk; 09-01-2014 at 11:31 PM.
arnyk is offline  
post #25 of 100 Old 09-02-2014, 08:25 AM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,190
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 125 Post(s)
Liked: 327
Quote:
Originally Posted by Garidy View Post
Unnecessary comment!

How would you feel if I referred to you in a derogatory manner?
My apologies. I did not mean to offend. I let my frustration get the better of me.
Garidy likes this.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is offline  
post #26 of 100 Old 09-02-2014, 09:08 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Quote:
Originally Posted by mcnarus View Post
If you do a SBT and fail to reliably hear a difference, it can be safely assumed that you would have similarly failed to hear a difference in a DBT.
Not at all. Imagine a scenario where I play the two tracks and keep telling you they sound the same. Arny does this with his listening tests where he prefaces them with not being beatable. That clearly biases the experiment and can result in the tester to for example give up quickly and not try to find any differences/critical segments. If I took away the proctor bias, then in DBT I may be able to find the differences.

Quote:
But if you do a SBT and get a statistically significant result, you cannot say that you reliably heard a difference. Maybe you did, or maybe the test administrator tipped you off, consciously or subconsciously.
Likewise, this is wrong too. As I explained to Arny when he post the very same response in WBF Forum, you absolutely can get reliable results out of single blind tests. And that the most insidious problem with these tests is the person who created the test. What they can do to cook the results easily dwarfs what a proctor can do that has the answer.

As a real example, when I was at Harman last, Dr. Sean Olive (president of Audio Engineering Society) ran through his "how to listen" test of whether one can hear EQ changes. There were a group of 10 to 15 of us. For the easiest levels 1 through 3 or so, everyone more or less knew the answers. Having had a bit of practice with their training tool, I was alone in keeping up with Sean above those levels. I forget the exact number but around level 6 or 7, I could no longer do that but Sean instantly gave the answer. Despite having the answer, Sean did not help me in any way getting more answers right. Or the other 10 to 15 people. Of course having the calm and quiet demeanor that Sean has helps a lot .

So no, we don't get to dismiss listening tests just because they are single blind. Calling them nearly the same as sighted is especially wrong. If this were so, I and the rest of the group could have kept up with Sean but clearly we could not. See my WBF forum thread for more real life examples.

A ton of research and development is done in the industry using single blind testing. Don't let these forum talking points steer you wrong. The real motive is to put down the data and single vs double is just a convenient excuse. The argument has no place in an informed conversation.
stereoeditor and acuity like this.

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #27 of 100 Old 09-02-2014, 09:25 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,387
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 763 Post(s)
Liked: 1178
Quote:
Originally Posted by amirm View Post
Not at all. Imagine a scenario where I play the two tracks and keep telling you they sound the same. Arny does this with his listening tests where he prefaces them with not being beatable.
The above seems to be based on someone's penchant for finding something that someone posted in a different context and a different time.

Quote:
That clearly biases the experiment and can result in the tester to for example give up quickly and not try to find any differences/critical segments. If I took away the proctor bias, then in DBT I may be able to find the differences.
Another technique to foul the well is to accompany one's comments about their DBTs with a litany of hypercritical critiques of just about anything related to the tests at hand. I've seen this played so well at another forum that nobody there seems to be brave enough to admit that they've taken 6 minutes to run a set of trials. At least a few people at AVS admit that they've tried.

Quote:
Likewise, this is wrong too. As I explained to Arny when he post the very same response in WBF Forum, you absolutely can get reliable results out of single blind tests.
One can do anything, but the relevant question is what gets done. After reading several spirited defenses of SBTs one wonders why FOOBAR2000 doesn't have a SBT mode! ;-)

If you can do better than a SBT (i.e., a DBT) then the DBT is what you should do. Period. MY comments about SBTs were in a different context, but context seems to be utterly meaningless in some people's eyes. When the goal is tar and feathers, why worry about context? ;-)

Quote:
And that the most insidious problem with these tests is the person who created the test. What they can do to cook the results easily dwarfs what a proctor can do that has the answer.
The role of proctors seems to be badly misunderstood. A unsupervised listener can wreak unlimited confusion especially if they are highly vocal.

Quote:
As a real example, when I was at Harman last, Dr. Sean Olive (president of Audio Engineering Society) ran through his "how to listen" test of whether one can hear EQ changes. There were a group of 10 to 15 of us. For the easiest levels 1 through 3 or so, everyone more or less knew the answers. Having had a bit of practice with their training tool, I was alone in keeping up with Sean above those levels. I forget the exact number but around level 6 or 7, I could no longer do that but Sean instantly gave the answer. Despite having the answer, Sean did not help me in any way getting more answers right. Or the other 10 to 15 people. Of course having the calm and quiet demeanor that Sean has helps a lot .
I don't have the resources of Harman and I don't get to instruct listeners in person, but in my recent jitter tests I also provided training samples that accomplished the same outcome. Of course someone seems to be so busy demonizing me that I get no credit for doing what I can.

Quote:
So no, we don't get to dismiss listening tests just because they are single blind.
Yup, Stereophile's and Analog Planet's SBTs related to Furutech CD demagnetizers are as valid as anything. Mike Fremer says so!
arnyk is offline  
post #28 of 100 Old 09-02-2014, 09:30 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,336
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 475 Post(s)
Liked: 831
Quote:
Originally Posted by arnyk View Post
Unfortunately obtaining the same results as another test doesn't globally validate the first one.

Nor did I say it did. I just said it worked and produced the same results.
FMW is offline  
post #29 of 100 Old 09-02-2014, 09:51 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Quote:
Originally Posted by arnyk View Post
The above seems to be based on someone's penchant for finding something that someone posted in a different context and a different time.
That someone is you Arny. Given the fact that we are discussing your positioning of blind tests, then it is extremely appropriate to post examples of "DBTs" that did not guard against experimenter bias:

Quote:
Originally Posted by arnyk View Post
Yes. Take the best audio system you can find. Take the best recordings you can find - recordings that sound great and also have significant content > 20 KHz, even > 35 KHz. Switch a 16 KHz brick wall filter in and out of the signal path. Nobody notices nuttin'

People say: "But I can hear pure tones at 21 KHz". Probably true. But that is without content at other frequencies masking it. Music is composed of many tones at many different frequencies. Masking in the upward direction frequency-wise is very strong.
Nobody notices nuttin'? You don't think you corrupted the outcome with that biased remark? How about the bit about masking?

We know as a matter fact that people can hear above the maximum bandwidth of 16 Khz which 32 K sampling represents. Yet you put forward a test, hoping by biasing it in advance, folks give up and produce negative outcome. And that worked initially! Here is what happened:

Quote:
Originally Posted by imagic View Post
Here's how I did on what should be the easiest test...

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/11 20:36:03

File A: E:\AVS\Foobar ABX\Jangling Keys\keys jangling band resolution limited 3216 2496.wav
File B: E:\AVS\Foobar ABX\Jangling Keys\keys jangling full band 2496.wav

20:36:03 : Test started.
20:37:08 : 00/01 100.0%
20:38:14 : 01/02 75.0%
20:39:19 : 01/03 87.5%
20:39:56 : 02/04 68.8%
20:40:17 : 02/05 81.3%
20:40:39 : 02/06 89.1%
20:41:13 : 02/07 93.8%
20:41:40 : 03/08 85.5%
20:42:09 : 03/09 91.0%
20:42:39 : 04/10 82.8%
20:42:55 : 05/11 72.6%
20:43:13 : 06/12 61.3%
20:44:03 : 06/13 70.9%
20:44:32 : 06/14 78.8%
20:45:55 : 07/15 69.6%
20:46:15 : 08/16 59.8%
20:46:31 : 08/17 68.5%
20:46:51 : Test finished.

----------
Total: 8/17 (68.5%)
In other words, he did "worse than chance." After a few back and forth discussing my positive results, Mark then produced these results:

Quote:
Originally Posted by imagic View Post
Laptop? Practice?
Well, I decided to give my laptop a try since Amir did so well using his. Lo and behold, I had little difficulty with the 16/32 key jangling test. Not quite perfect, but I suspect a bit more practice would get me up to perfect.

The results speak for themselves; I found a critical segment that revealed an audible difference. I've had some practice, which helped—just as Amir suggested. Now, I can pass an ABX test I previously failed. I'll tackle the 16/44 test next. Oh, and it was a piece of cake to pick out the differences in the 16/16 and 22/16 tests.

foo_abx 1.3.4 report
foobar2000 v1.3.3
2014/07/19 11:26:49

File A: C:\Users\mark_000\Downloads\keys jangling band resolution limited 3216 2496.wav
File B: C:\Users\mark_000\Downloads\keys jangling full band 2496.wav

11:26:49 : Test started.
11:27:29 : 00/01 100.0%
11:28:58 : 00/02 100.0%
11:29:46 : 00/03 100.0%
11:29:59 : 01/04 93.8%
11:30:06 : 01/05 96.9%
11:30:16 : 02/06 89.1%
11:30:26 : 03/07 77.3%
11:30:34 : 04/08 63.7%
11:30:45 : 05/09 50.0%
11:31:00 : 06/10 37.7%
11:31:10 : 07/11 27.4%
11:31:29 : 08/12 19.4%
11:31:41 : 09/13 13.3%
11:32:05 : 10/14 9.0%
11:32:20 : 10/15 15.1%
11:32:30 : 11/16 10.5%
11:32:41 : 12/17 7.2%
11:32:52 : 13/18 4.8%
11:33:07 : 13/19 8.4%
11:33:16 : 14/20 5.8%
11:33:28 : 15/21 3.9%
11:33:40 : 16/22 2.6%
11:33:58 : 17/23 1.7%
11:34:12 : 18/24 1.1%
11:34:25 : Test finished.

----------
Total: 18/24 (1.1%)
Essentially perfect results. As the "science" would have predicted. The first tests he ran were all biased by Arny and countless other forum posts backing what he has been saying. So he didn't try hard at all. His expectation bias was that he couldn't hear the difference and produced the same. Fact that the test was "double blind" did not help whatsoever in isolating the tester from the experiment creator.

Net, net, you can't read the book by its cover. There is no shortcut to validity or lack thereof of a test based on three letter acronyms.

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #30 of 100 Old 09-02-2014, 10:03 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,375
Mentioned: 4 Post(s)
Tagged: 0 Thread(s)
Quoted: 981 Post(s)
Liked: 547
Quote:
Originally Posted by arnyk View Post
Another technique to foul the well is to accompany one's comments about their DBTs with a litany of hypercritical critiques of just about anything related to the tests at hand. I've seen this played so well at another forum that nobody there seems to be brave enough to admit that they've taken 6 minutes to run a set of trials. At least a few people at AVS admit that they've tried.
Hypercritical about DBT? No, I am hypercritical of campaign slogans put forward as "science." Single blind is the same as sighted test? You have to be kidding me.

As to the 6 minutes, I have spent far more than that running your DBTs. You on the other hand, have hardly run any of them. Why won't you post the results of your "jiter" DBT Arny? Does it take you more than 6 minutes to run them? I ran four sets of them, all with positive outcome. So it was more than "trying." Set the example please before complaining about others.

Quote:
One can do anything, but the relevant question is what gets done. After reading several spirited defenses of SBTs one wonders why FOOBAR2000 doesn't have a SBT mode! ;-)
There is no defense of "SBTs." There is an argument and a strong one, against the statement that it is similar to sighted tests.

Quote:
If you can do better than a SBT (i.e., a DBT) then the DBT is what you should do. Period. MY comments about SBTs were in a different context, but context seems to be utterly meaningless in some people's eyes. When the goal is tar and feathers, why worry about context? ;-)
The comment you made deserved the tar and feathers:
Quote:
Originally Posted by arnyk View Post
When you listen to these files, remember that "Blind Test" generally means "Single Blind Test" and that a single blind test is just a sighted evaluation with the some of the appearance of a DBT , but not the real thing.
Single blind test is just a sighted evaluation? Not the real thing? And you say the context has changed?

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
Closed Thread Audio Theory, Setup, and Chat

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off