A Different Take on Blind Audio Testing - AVS Forum
Forum Jump: 
 18Likes
  • 8 Post By Scott Wilkinson
  • 1 Post By Divergent9999
  • 2 Post By cunim
  • 2 Post By mike_carton
  • 2 Post By audioguy
  • 2 Post By Utopianemo
  • 1 Post By tgm1024
 
Thread Tools
post #1 of 21 Old 08-29-2014, 01:46 PM - Thread Starter
AVS Special Member
 
Scott Wilkinson's Avatar
 
Join Date: Sep 2001
Location: Burbank, CA
Posts: 1,295
Mentioned: 7 Post(s)
Tagged: 0 Thread(s)
Quoted: 306 Post(s)
Liked: 1084
A Different Take on Blind Audio Testing



Jon Iverson, audiophile and web monkey for Stereophile.com, SoundandVision.com, InnerFidelity.com, AudioStream.com, and AnalogPlanet.com, talks about his interpretation of blind audio testing and what it does—and does not—reveal about the performance of audio equipment, recordings, and the participants in the test. He also discusses his own experiences listening for differences in DACs, the importance of listener training, the music-recording process, the AVS/AIX high-resolution audio experiment, answers to chat room questions, and more.


Like AVS Forum on Facebook
Follow AVS Forum on Twitter
+1 AVS Forum on Google+

Scott Wilkinson
AVS Editor
Scott Wilkinson is offline  
Sponsored Links
Advertisement
 
post #2 of 21 Old 08-30-2014, 12:21 AM
Senior Member
 
catonic's Avatar
 
Join Date: Feb 2008
Posts: 257
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 44 Post(s)
Liked: 69
Thanks again Scott, a very informative show by someone who understands the practical science of the issues under discussion.
catonic is offline  
post #3 of 21 Old 08-30-2014, 12:47 AM
AVS Special Member
 
tenthplanet's Avatar
 
Join Date: Feb 2006
Location: North of Mexico, South of Oregon, Not as far east as Vegas
Posts: 1,489
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 171
Excellent show Scott, please have Jon on again in the future.

"Bring out yer dead!".."Wait I'm not dead yet!"..(Sound Austrian here) "WRONG !!" (You know what happens next..)
tenthplanet is online now  
post #4 of 21 Old 08-30-2014, 06:42 PM
Member
 
Divergent9999's Avatar
 
Join Date: Apr 2014
Posts: 39
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 22
Thank you Scott for this excellent, well-balanced podcast. Your guests are always informative.
Cerumen likes this.
Divergent9999 is offline  
post #5 of 21 Old 08-30-2014, 07:27 PM
AVS Special Member
 
Jinjuku's Avatar
 
Join Date: Mar 2007
Location: Louisville, KY
Posts: 2,296
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 30 Post(s)
Liked: 112
Thanks Scott. I agree 100% with the assessment that we are testing peoples ability to discern differences.

He made a great point about statistical difference. If you have a room of 1000 people and 2 ace the SBT/DBT/AB-X or other discrimination you have 0.2% of a population that follows. It's not buried in the data. There is no scrum. You have a hard number. Now repeat this 1000N often enough then extrapolation can take place.

When comparing amps or any other gear: Manufacturers can make items purposefully sound different and then market that difference.

The other thing to consider: Of the two people at the 1989 event: Did they have a preference? Did that preference follow price of the amp even if they didn't know the DUT? Would they use the effusive term: "Night and Day Difference"?

An audiophile likes to talk about how much they spent and how good it sounds.

A DIY'er likes to talk about how little they spent and how good it sounds.


Last edited by Jinjuku; 08-30-2014 at 07:31 PM.
Jinjuku is offline  
post #6 of 21 Old 08-31-2014, 12:36 PM
AVS Addicted Member
 
LTD02's Avatar
 
Join Date: Oct 2003
Posts: 16,733
Mentioned: 17 Post(s)
Tagged: 0 Thread(s)
Quoted: 496 Post(s)
Liked: 1122
bravo.

Listen. It's All Good.
LTD02 is online now  
post #7 of 21 Old 08-31-2014, 07:49 PM
Newbie
 
Join Date: Jul 2014
Posts: 13
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 2
Loved the milkshake analogy for DACs!

Jeff Perrin
Surround Recording Artist
Home Theater Ear Candy App - Dramatic 5.1 music for iOS and the Apple TV
http://AppStore.com/HomeTheaterEarCandy
JeffPerrin is offline  
post #8 of 21 Old 09-01-2014, 02:33 PM
AVS Special Member
 
WhskyTangoFoxtrt's Avatar
 
Join Date: Nov 2008
Location: Fort Worth, TX
Posts: 1,349
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 22 Post(s)
Liked: 37
Very cool. Is there a transcript available?
WhskyTangoFoxtrt is offline  
post #9 of 21 Old 09-01-2014, 02:57 PM
Newbie
 
cunim's Avatar
 
Join Date: Sep 2008
Posts: 6
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 12
Where do I start?

Quote:
Originally Posted by Scott Wilkinson View Post

Jon Iverson, audiophile and web monkey for Stereophile.com, SoundandVision.com, InnerFidelity.com, AudioStream.com, and AnalogPlanet.com, talks about his interpretation of blind audio testing and what it does—and does not—reveal about the performance of audio equipment, recordings, and the participants in the test. He also discusses his own experiences listening for differences in DACs, the importance of listener training, the music-recording process, the AVS/AIX high-resolution audio experiment, answers to chat room questions, and more.

https://www.youtube.com/watch?v=p6kC5vck2JA

Like AVS Forum on Facebook
Follow AVS Forum on Twitter
+1 AVS Forum on Google+
Jon, I sense your pain but there is no need for it. You have a fundamental misunderstanding of statistics, experimental design, and sampling theory (I used to teach these). Please don't take offence. Why study a fairly dry discipline (experimental design) if it is not needed for your day to day function? An audio reviewer need not understand statistical methodology or the nature of a sample before doing subjective evaluation of components or music. In subjective evaluation, everyone knows you are making statements about what YOU experience. To the extent that you have proven yourself to be a good judge of quality (lots of people agree with you), your statements have relevance to the state of the world in general - what we call external validity. Hence, there can be a demonstrable link between what one skilled observer reports and the state of the real world. That is what underlies the entire field of audiophile reviewery.


However, the present discussion goes beyond subjective observations. Here you are making statements about objective testing and experiments. You suggest that a limited sample (the people in the room at the moment) corrupts your belief in any observation that amp A is "really" different than amp B for example. Take heart. It is possible to make such a statement and to give a formal definition of your confidence in that statement. I would suggest contacting a friendly audio engineer or someone with a background in psychophysics and having a chat. Failing that, take an on line course in xperimental design. It would give you everything you need to sort out this foolishness. Basically, if your sample is well constructed and if your methods are appropriate to the experiment or survey in hand, the issues you raise are non-issues. Hence, back to my point about there being no need for pain regarding the validity of blind testing. Whether it is valid or not is dependent upon how it is done, not upon some fundamental flaw in the concept. Of course you are testing only the people in your sample - there is no one else there to test. The trick is to make sure it is a representative sample (most simply by being randomly sampled but lots of other ways) so that it tells you something about the real world.


Rant over.
mike_carton and lovinthehd like this.
cunim is offline  
post #10 of 21 Old 09-02-2014, 01:50 AM
Senior Member
 
catonic's Avatar
 
Join Date: Feb 2008
Posts: 257
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 44 Post(s)
Liked: 69
^ ^ Those who can do, those who can't teach. Please don't take offence.
catonic is offline  
post #11 of 21 Old 09-02-2014, 07:13 AM
Member
 
mike_carton's Avatar
 
Join Date: Apr 2006
Posts: 55
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 17
Quote:
Originally Posted by catonic View Post
^ ^ Those who can do, those who can't teach. Please don't take offence.
The quality of my online life has improved since I started using the Ignore list. On you go.
wolfereeno and andyc56 like this.
mike_carton is offline  
post #12 of 21 Old 09-02-2014, 09:24 AM
AVS Special Member
 
audioguy's Avatar
 
Join Date: Jun 2001
Location: Not far from Atlanta - but far enough!
Posts: 3,367
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 279 Post(s)
Liked: 248
I stopped listening after the conversation wandered away from blind listening. But Scott asked the question: "If only one person can consistently tell the difference, does that not mean there is a difference?" But Mr Iverson kept going back to his position that it was more about the testers rather than the equipment. I believe the answer to Scott's question has to be "yes".

Mr. Iverson's comments on DAC differences sort of sums up my position on virtually most electronics. If DAC A has very very subtle differences over DAC B AND it takes multiple hours to eventually hear the difference AND the next time you listen you have to start all over again to hear the difference, I have gotten to the position that the differences must be too small for me to spend time (and money) worrying about. Or said in another way: If product A sounds different than product B but I can't very easily hear the difference, there is no difference.

This session notwithstanding, I will continue to use blind testing as I evaluate electronic products. I am not trying to demonstrate that there are or are not differences - only if I can or can not hear them.
Utopianemo and lovinthehd like this.

Even when I'm not having a particularly good day, I am at least having a day!

New Theater 
 
Build Thread
audioguy is offline  
post #13 of 21 Old 09-02-2014, 10:45 AM
Newbie
 
dinomite's Avatar
 
Join Date: Apr 2014
Posts: 6
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 12
Quote:
Originally Posted by cunim View Post
However, the present discussion goes beyond subjective observations. Here you are making statements about objective testing and experiments. You suggest that a limited sample (the people in the room at the moment) corrupts your belief in any observation that amp A is "really" different than amp B for example. Take heart. It is possible to make such a statement and to give a formal definition of your confidence in that statement. I would suggest contacting a friendly audio engineer or someone with a background in psychophysics and having a chat. Failing that, take an on line course in xperimental design. It would give you everything you need to sort out this foolishness. Basically, if your sample is well constructed and if your methods are appropriate to the experiment or survey in hand, the issues you raise are non-issues. Hence, back to my point about there being no need for pain regarding the validity of blind testing. Whether it is valid or not is dependent upon how it is done, not upon some fundamental flaw in the concept. Of course you are testing only the people in your sample - there is no one else there to test. The trick is to make sure it is a representative sample (most simply by being randomly sampled but lots of other ways) so that it tells you something about the real world.


Rant over.
I guess I'm missing something here. You're saying that you can take the subjectivity out of the test, by choosing the right people?
dinomite is offline  
post #14 of 21 Old 09-02-2014, 10:58 AM
AVS Special Member
 
lovinthehd's Avatar
 
Join Date: Dec 2007
Location: OROR
Posts: 6,830
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 298 Post(s)
Liked: 843
Quote:
Originally Posted by catonic View Post
^ ^ Those who can do, those who can't teach. Please don't take offence.
Thought it was "those who can't do, teach; those who can't teach, teach gym" ?

lovinthehd is online now  
post #15 of 21 Old 09-02-2014, 04:19 PM
Newbie
 
cunim's Avatar
 
Join Date: Sep 2008
Posts: 6
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 4 Post(s)
Liked: 12
Quote:
Originally Posted by dinomite View Post
I guess I'm missing something here. You're saying that you can take the subjectivity out of the test, by choosing the right people?

Actually, that is close to correct if you substitute the term "error" for the term "subjectivity". We can minimize the chance that our test gives us an erroneous result (fails to detect a real effect or detects a false effect), even if we are recording subjective data.


My point is that there are methods for doing all this. There is nothing remotely mysterious about it - it is first year undergraduate material in psychology, sociology, many aspects of biology, marketing, political polling etc.


Yes, there is a whole library of methods that allow you to ask whether people can hear a difference between components - and even how big the difference has to be before it can be heard. Sadly, doing quality tests is much harder than doing informal ones. So much easier to use measurements (which we use to "operationally define" the good) or reviewers or our own listening. In fact, a combination of these works pretty well because people keep buying new gear and liking what they buy. Clearly, these people hear a difference.


So it does come down to picking the right people. In making inferences about a population, we need to use formal methods to define the population (people in general, audiophiles, experienced audiophiles, whatever), select the samples, and present the test. Then the uncertainties that Jon raises are minimized.


Or not. We can just realize this is a hobby and have fun with it. Maybe you believe you hear a difference between two brands of speaker cable. Believing that makes you happy. Maybe you could demonstrate the cable sensitivity on a proper blind test or maybe you couldn't. For example, I can't hear cables though I hear amps and preamps pretty reliably on informal tests. So, I use low cost cables and great amps/preamps. Works for me, and I agree with Jon when he implies that the differences that are real are the ones that matter to you. Just don't bash formal testing methods (such as blind testing) unless you understand the theory behind them.
cunim is offline  
post #16 of 21 Old 09-02-2014, 05:14 PM
Member
 
Join Date: Jul 2014
Location: San Diego, CA
Posts: 48
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 26 Post(s)
Liked: 6
I thoroughly enjoyed another TWiT Home Theater Geeks Netcast.
Jon Iverson is special and I agree with most of what he
stated here. I've been involved with a number of A/B/X tests.
Certainly the group consisted of "Golden Ears" and pros
from the Audio recording industry. We pretty much came up
with the same conclusions Jon articulated so well here.
A/B/X does NOT tell you what's better then the other.
It simply tells you if there is a perceptive difference
between two items under test. We did the classic 16ga zip
cord vs $10k speaker wire A/B/X testing and under normal
conditions it was difficult to tell the difference.
I also agree with Jons observation that it's a learned
process that has to be relearned after time goes by.
wilcal is offline  
post #17 of 21 Old 09-03-2014, 08:17 AM
Senior Member
 
Utopianemo's Avatar
 
Join Date: May 2007
Location: Portland, OR
Posts: 334
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 22 Post(s)
Liked: 39
Quote:
Originally Posted by wilcal View Post
I thoroughly enjoyed another TWiT Home Theater Geeks Netcast.
Jon Iverson is special and I agree with most of what he
stated here. I've been involved with a number of A/B/X tests.
Certainly the group consisted of "Golden Ears" and pros
from the Audio recording industry. We pretty much came up
with the same conclusions Jon articulated so well here.
A/B/X does NOT tell you what's better then the other.
It simply tells you if there is a perceptive difference
between two items under test. We did the classic 16ga zip
cord vs $10k speaker wire A/B/X testing and under normal
conditions it was difficult to tell the difference.
I also agree with Jons observation that it's a learned
process that has to be relearned after time goes by.
Jon's most important point, in my opinion, is that blind testing is primarily to test the person, not the product.

There's a bit of a disconnect occurring when audio buffs assert that blind/double-blind/AB-X testing is the only way to properly assess differences in audio products. The subtext is that they fundamentally don't believe there is a discernible difference between a sampling of amps of reasonable quality, or a sampling of audio cables, etc. The desire for such testing is more of an assertion that since there is presumably no difference between such products, then reviewers who lavish flowery praise over certain (expensive)products must be willingly or unwittingly deceiving themselves and their reader base. Such testing in this case is really more about the desire to show the reviewer is either a liar or a fool.

Whether or not this happens is another matter entirely; but I'm glad Jon at least recognizes the subterfuge. I personally tire of hearing reviewers go into ecstatic prose over exorbitantly expensive products, especially cabling. But I'm not so callous as to claim that there is no discernible difference; I just could care less because it has no real impact on me or what my budget allows.
catonic and JeffPerrin like this.
Utopianemo is offline  
post #18 of 21 Old 09-03-2014, 06:30 PM
Senior Member
 
catonic's Avatar
 
Join Date: Feb 2008
Posts: 257
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 44 Post(s)
Liked: 69
^ ^ A sensible response, IMO.
catonic is offline  
post #19 of 21 Old 09-03-2014, 09:15 PM
AVS Special Member
 
NorthSky's Avatar
 
Join Date: Jan 2013
Location: Canada - West Island: Vancouver, South Direction: Go East
Posts: 3,700
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 1584 Post(s)
Liked: 500
Quote:
Originally Posted by Utopianemo View Post
Jon's most important point, in my opinion, is that blind testing is primarily to test the person, not the product.

There's a bit of a disconnect occurring when audio buffs assert that blind/double-blind/AB-X testing is the only way to properly assess differences in audio products. The subtext is that they fundamentally don't believe there is a discernible difference between a sampling of amps of reasonable quality, or a sampling of audio cables, etc. The desire for such testing is more of an assertion that since there is presumably no difference between such products, then reviewers who lavish flowery praise over certain (expensive)products must be willingly or unwittingly deceiving themselves and their reader base. Such testing in this case is really more about the desire to show the reviewer is either a liar or a fool.

Whether or not this happens is another matter entirely; but I'm glad Jon at least recognizes the subterfuge. I personally tire of hearing reviewers go into ecstatic prose over exorbitantly expensive products, especially cabling. But I'm not so callous as to claim that there is no discernible difference; I just could care less because it has no real impact on me or what my budget allows.
I really like your reply; it sounds/resonates right.
NorthSky is offline  
post #20 of 21 Old 09-04-2014, 08:19 AM
AVS Special Member
 
tgm1024's Avatar
 
Join Date: Dec 2010
Posts: 6,602
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 530 Post(s)
Liked: 867
This reminds me of something. This audio salesman once confided in me that if he really wanted to sell an amp over some other model, he'd simply raise the volume on it ever so slightly. He said this worked "every time".
Shadowed likes this.

Violence is the last refuge of the incompetent. Unless, of course, it's to keep someone from creating a phone video in portrait mode, in which case it's a pretty good first step. Portrait mooks: KNOCK IT OFF.
tgm1024 is offline  
post #21 of 21 Old 09-04-2014, 06:09 PM
Senior Member
 
Gaugster's Avatar
 
Join Date: Feb 2005
Location: Chicago, IL
Posts: 325
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 31 Post(s)
Liked: 16
Liked the video.

On stage 4 of the Philips Golden Ear testing.

Good times!

Gaugster
Gaugster is offline  
Reply AVS Forum Podcasts

Tags
frontpage , Home Theater Geeks

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off