Validty of blind testing - Page 4 - AVS Forum
Forum Jump: 
 
Thread Tools
post #91 of 355 Old 02-04-2014, 07:05 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 4,746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 230 Post(s)
Liked: 697
Quote:
Originally Posted by arnyk View Post

Tradition is a very strong influence on people's lives. For maybe 7 decades people have been told that "new improved" audio gear "sounds better". It's tradition.

It's marketing. Audiophiles are smart enough to understand that.
FMW is online now  
Sponsored Links
Advertisement
 
post #92 of 355 Old 02-04-2014, 07:29 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by FMW View Post

Personally I think it is the concept of golden ears. I think audiophiles believe they can hear better than other people and therefore have a feeling of superiority over others. It isn't about equipment all, perhaps. It is about their own skill level. They are furious that their hearing is biased just like everyone else's. It degrades that feeling of superiority.

I'm confused by that sentiment if it is true. Since high end audiophiles are generally higher up the intelligence and income scale, one would think they would look at thing with more analytical than emotional approach. The bias controlled listening test is an ear opener that every audiophile should experience. It is such a shame that it is such a fussy and difficult thing to do. I see no problem with wanting to own a $3500 CD player any more than owning a Patek Phillippe watch. It is a lovely thing to own. Why does it have to sound better?
I think your sample size is too small to arrive at the above generalization. Unless you tell me that you have tested 1000 audiophiles, you don't know if it is or isn't possible for them to have better hearing than you. We, i.e. my team at Microsoft, have done that testing. While it is true that majority of self-appointed audiophiles have no better ability to hear non-linear distortions, there are exceptions. We for example found people who could outperform even our expert listeners with no training at all.

First let's review how good expert listeners can be. This is from Harman testing a sample of speakers with expert and non-expert listeners:

TrainedvsUntrained.png

Clearly the expert listeners have way above ability to detect distortion and rate it down. B is B&W speaker and M is Martin Logan. If you look at their frequency response, we know that they have serious issues that the expert listeners could detect readily. While their preference as a whole was no different than average population, they clearly could detect and pinpoint problems a heck of a lot easier. If we had shrunk the differences down there would be a point where they could hear distortions that the average person would not. This doesn't occur in speakers since they always sound so different but in other audio matters, it may.

Harman gives away a tool that is used to train people. I have taken the test without training, and with some training and in person compared my assessment to Sean. He could outperform me in hearing frequency anomalies with ease in double blind testing although my bit of training allowed me to pull way ahead of average people. So let's agree that trained listeners do better than average people in this context (one of the few instances where I disagree with Dr. Toole/Olive).

In our large scale testing at Microsoft of audiophiles, we did find an occasional person that outperformed even our best listeners. I was one of those "best listeners" so when I say that, it doesn't come easy smile.gif. Here is the important part: we identified the technical flaw and fixed it and that person no longer heard the problem that the rest of us could not.

Here is the key point: in the kind of double blind tests you all care about, we rarely if ever know the answer. So we believe any outcome that the test shows. Note that in the example I gave, such was not the case. We could have easily said that the odd person was wrong. But we had created the technology and could based on his assessment pinpoint the problem, fix it, and then retest. It was that deep knowledge of technology that allowed us to detect that the problem was real. In whatever testing you did, you didn't have such information so it is entirely possible that you convinced yourself of the wrong conclusion, both about what you tested and importantly, about other people's abilities.

We have a similar example of this in Swedish LTS testing of Bryston amplifier. In blind testing, they found out that the amp colored the input sound. They gave that feedback to Bryston, they made some circuit changes and the second blind test found it to be transparent to its input: http://www.4audio.rs/upload/File/Bryston/Bryston%20review%2049.pdf

Pre-fix by Bryston:

"First Bypass Listening Test
And now the anti climax reaches its maximum – the Bryston amplifier was easily identified by the shift in timbre – just in the way we thought – it put some colorations to the music that made the music sound somewhat darker than with a neutral amp. Furthermore, we identified it as slightly dynamically restrained, but from experience we know that this type of coloration can have its origin in the tonal character, even when it doesn’t sound like it is about timbre."


This led to conversation with Bryston:

"Altogether we probably spoke five or ten minutes over the phone, and I tried to explain as well as I could my view on the problems and what I believed would be suitable solutions. It all ended up with them redesigning parts of the amp according to my suggestions. Not only this sample, but all amps in running production from that date."

Re-testing blind resulted in this outcome:

“To sum it up, normally there are lots of views, ideas and opinions regarding the character of the tested amplifier after the open listening....We were sitting in open listening for well over one hour, and no one mentioned a single word about any differences they either imagined or heard. Actually, that’s the first time ever something like that has happened. We kept on listening without any changes. We switched between B and A, sometimes rapidly, sometimes with longer intervals. We tried having the music running while switching, and to play shorter loops and switch so that identical sequence was heard on B and A.”

If it were the present crowd performing blind testing of amps, what would have been the possibility of them catching this issue let alone getting Bryston to fix it? They all believe there is no difference between amps in blind tests, right? But here we are.

What is the difference, these guys created a better and more revealing test. They test an amplifier's input against its (attenuated) output. That is what we are interested in, right? That is, an amp coloring its input or not. Testing random amp A against random amp B at heaven knows what volume and what content is not remotely as good as what they have done.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #93 of 355 Old 02-04-2014, 07:42 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by arnyk View Post

We're back to the same old game - demonize DBTs because they don't meet someone's personal standards for perfection, when the world is full of people basing all kinds of audio decisions on the most flawed evaluations one can imagine - the non-level matched, non-time-synched, sighted evaluation.
Speaking for myself, I am not at all demonize double blind tests whatsoever. What I am highly critical of are people believing their results without spending any energy, or understanding the limits of their technology knowledge. I can compare water to coke in a test and if I say it is double blind, folks would believe that the two taste the same smile.gif. It is almost that bad. Folks like yourself work backward: you focus on the answer and if it agrees with your views of audio, you don't look at the test critically at all. Every sin could have existed in there and it would have been just fine. The answer is right so the question must have been also. I gave the example earlier of how the flaws in Mayer and Moran tests of high resolution audio was found not by objectivists, but the camp that we call "non-scientific." The non-scientific folks thought the results disagreed with their ears, performed spectrum analysis and found out their ears were right and the test wrong!

That is how we lose credibility. I am a super strong supporter of the value of double blind tests. But I judge them as well as I judge sighted tests. I don't trust any just because it has the words "double blind" next to it. As you said yourself, all tests are flawed. You must start with finding those flaws first, and then looking at the outcome.

You have a vested interest in defending double blind tests at all cost. It is what you have built your reputation on. I don't have that to defend. That makes it easier for me to critical of any test, blind or not. It is this lack of transparency that gets me to argue with you so much even though at some level we are in the same camp. Audiophiles are smart and can smell a rat. They know about your predisposition and the more you re-enforce that, the less they will listen to you. That then questions the very foundation of what you are trying to accomplish here: the people you are most upset at, don't care one bit what you or the other vocal members here have to say!

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #94 of 355 Old 02-04-2014, 07:58 AM
AVS Special Member
 
BruceD's Avatar
 
Join Date: Oct 2000
Location: Silicon Valley, CA USA
Posts: 1,064
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 1 Post(s)
Liked: 29
Quote:
Originally Posted by amirm View Post

. . . . . . I am a super strong supporter of the value of double blind tests. But I judge them as well as I judge sighted tests. . . . ;.

You miss the point entirely, sighted tests have no validity whatsoever as tests for audio comparisons, they are simply an exercise in displaying someone's personal agenda, especially people like you who have a vested interest in selling high dollar audio equipment.
67jason, andyc56 and Jack D Ripper like this.
BruceD is offline  
post #95 of 355 Old 02-04-2014, 08:10 AM
Senior Member
 
R Swerdlow's Avatar
 
Join Date: Nov 2007
Location: Gaithersburg, MD
Posts: 371
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 32
Some quotes are needed here:

The first principle of scientific inquiry is that you must not fool yourself – and that you are the easiest person to fool. – Richard Feynman

Knowledge of the products that are being evaluated by subjective testing is generally understood to be a powerful source of psychological bias. In scientific tests of many kinds of human perception, even wine tasting, considerable effort is expended to hide the identity of what is being subjectively evaluated. In audio, though, things are more relaxed, and people, who are otherwise serious, persist in the belief that they are immune to the influence of such factors as price, size, brand, etc. In some of these so-called great debate issues, such as whether amplifiers, wires, and the like have an audible effect, there are those who claim that disguising a product’s identity actually prevents listeners from hearing differences that are in the range of extremely small to inaudible. That debate shows no signs of slowing down. – Floyd E. Toole

Another from FE Toole (in Sound Reproduction: The Acoustics and Psychoacoustics of Loudspeakers and Rooms, by Floyd E. Toole, page 357)

A wide spread belief among audio professionals is that they are immune to the influences of brand, price, appearance, and so on. They persist in conducting listening evaluations with the contending products in full view. This applies to persons in the recording industry, audio journalists/reviewers, and loudspeaker engineers. As this is being written, the 45th anniversary issue of Sterophile magazine (November 2007) arrived. In John Atkinson’s editorial, he interviewed J. Gordon Holt, the man who created the magazine. Holt commented as follows:

Audio as a hobby is dying, largely by its own hand. As far as the real world is concerned, high-end audio lost its credibility during the 1980s, when it flatly refused to submit to the kind of basic honesty controls (double-blind testing, for example) that had legitimized every other serious scientific endeavor since Pascal. [This refusal] is a source of endless derisive amusement among rational people and of perpetual embarrassment for me, because I am associated by so many people with the mess my disciples made of spreading my gospel.

And finally, from myself wink.gif:

Blind testing of anything involving human perception does not guarantee a valid test, but sighted testing guarantees an invalid test.
Chu Gai and FreeFire like this.
R Swerdlow is offline  
post #96 of 355 Old 02-04-2014, 08:16 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,275
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 674 Post(s)
Liked: 1142
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by arnyk View Post

We're back to the same old game - demonize DBTs because they don't meet someone's personal standards for perfection, when the world is full of people basing all kinds of audio decisions on the most flawed evaluations one can imagine - the non-level matched, non-time-synched, sighted evaluation.

Speaking for myself, I am not at all demonize double blind tests whatsoever.

To meet a reasonable standard for equity, a reasonable person might wish to see equally harsh and detailed criticism of sighted evaluations, as has been ladled out on top of DBTs in the post I was responding to.

Here's an example what appears to be a claim of validity for a sighted evaluation:

http://www.avsforum.com/t/1125344/jim-burns-are-some-blu-ray-players-better-on-hdmi-hd-picture-quality/150#post_15990901
Quote:
Originally Posted by AmirM 
"Many people, myself included, can hear something that is only explained by jitter. When I change S/PDIF cables, I hear the difference even with high-end DACs. Hard to imagine anything else being responsible for the difference."

Where is the criticism of this apparently sighted evaluation published?
Quote:
What I am highly critical of are people believing their results without spending any energy, or understanding the limits of their technology knowledge.

And we can see equal time/equal energy highly critical posting about sighted evaluations where? ;-)
arnyk is offline  
post #97 of 355 Old 02-04-2014, 08:27 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,275
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 674 Post(s)
Liked: 1142
Quote:
Originally Posted by amirm View Post


You have a vested interest in defending double blind tests at all cost.

Kinda sorta. I have a lot of vested interests in various issues in audio, DBTs being one of them.

However I am not the initiator of audio DBTs as an issue in audio. In my mind the late Dan Shanefield PhD gets the credit for that.

While one form of audio DBTs is credited to me (ABX) there are many other forms that are perhaps more widely used that were devised by other people. For example I knew nohing about MSHRA or ABC/hr until I read about them in public documents.
Quote:
It is what you have built your reputation on.

I suspect that my reputation in audio is a bit broader than that.
Quote:
I don't have that to defend.

Hmm, you've got an active apparently profit-making business that you founded, right?
Quote:
That makes it easier for me to critical of any test, blind or not.

Let's see how you defend your past hearty advocacy of sighted evaluations of subtle audio effects.
Quote:
It is this lack of transparency that gets me to argue with you so much even though at some level we are in the same camp.

That level has to be pretty darn remote, all things considered! ;-)
Quote:
Audiophiles are smart and can smell a rat. They know about your predisposition and the more you re-enforce that, the less they will listen to you.

I don't think that having a predispostion towards truth is a deficit in the market for ideas that I work.
Quote:
That then questions the very foundation of what you are trying to accomplish here: the people you are most upset at, don't care one bit what you or the other vocal members here have to say!

Just another example of what some might interpret as a belief that truth is up for a popularity vote.
arnyk is offline  
post #98 of 355 Old 02-04-2014, 08:52 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 4,746
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 230 Post(s)
Liked: 697
Quote:
Originally Posted by amirm View Post

I think your sample size is too small to arrive at the above generalization. Unless you tell me that you have tested 1000 audiophiles, you don't know if it is or isn't possible for them to have better hearing than you. We, i.e. my team at Microsoft, have done that testing. While it is true that majority of self-appointed audiophiles have no better ability to hear non-linear distortions, there are exceptions. We for example found people who could outperform even our expert listeners with no training at all.

First let's review how good expert listeners can be. This is from Harman testing a sample of speakers with expert and non-expert listeners:

TrainedvsUntrained.png

Clearly the expert listeners have way above ability to detect distortion and rate it down. B is B&W speaker and M is Martin Logan. If you look at their frequency response, we know that they have serious issues that the expert listeners could detect readily. While their preference as a whole was no different than average population, they clearly could detect and pinpoint problems a heck of a lot easier. If we had shrunk the differences down there would be a point where they could hear distortions that the average person would not. This doesn't occur in speakers since they always sound so different but in other audio matters, it may.

Harman gives away a tool that is used to train people. I have taken the test without training, and with some training and in person compared my assessment to Sean. He could outperform me in hearing frequency anomalies with ease in double blind testing although my bit of training allowed me to pull way ahead of average people. So let's agree that trained listeners do better than average people in this context (one of the few instances where I disagree with Dr. Toole/Olive).

In our large scale testing at Microsoft of audiophiles, we did find an occasional person that outperformed even our best listeners. I was one of those "best listeners" so when I say that, it doesn't come easy smile.gif. Here is the important part: we identified the technical flaw and fixed it and that person no longer heard the problem that the rest of us could not.

Here is the key point: in the kind of double blind tests you all care about, we rarely if ever know the answer. So we believe any outcome that the test shows. Note that in the example I gave, such was not the case. We could have easily said that the odd person was wrong. But we had created the technology and could based on his assessment pinpoint the problem, fix it, and then retest. It was that deep knowledge of technology that allowed us to detect that the problem was real. In whatever testing you did, you didn't have such information so it is entirely possible that you convinced yourself of the wrong conclusion, both about what you tested and importantly, about other people's abilities.

We have a similar example of this in Swedish LTS testing of Bryston amplifier. In blind testing, they found out that the amp colored the input sound. They gave that feedback to Bryston, they made some circuit changes and the second blind test found it to be transparent to its input: http://www.4audio.rs/upload/File/Bryston/Bryston%20review%2049.pdf

Pre-fix by Bryston:

"First Bypass Listening Test
And now the anti climax reaches its maximum – the Bryston amplifier was easily identified by the shift in timbre – just in the way we thought – it put some colorations to the music that made the music sound somewhat darker than with a neutral amp. Furthermore, we identified it as slightly dynamically restrained, but from experience we know that this type of coloration can have its origin in the tonal character, even when it doesn’t sound like it is about timbre."


This led to conversation with Bryston:

"Altogether we probably spoke five or ten minutes over the phone, and I tried to explain as well as I could my view on the problems and what I believed would be suitable solutions. It all ended up with them redesigning parts of the amp according to my suggestions. Not only this sample, but all amps in running production from that date."

Re-testing blind resulted in this outcome:

“To sum it up, normally there are lots of views, ideas and opinions regarding the character of the tested amplifier after the open listening....We were sitting in open listening for well over one hour, and no one mentioned a single word about any differences they either imagined or heard. Actually, that’s the first time ever something like that has happened. We kept on listening without any changes. We switched between B and A, sometimes rapidly, sometimes with longer intervals. We tried having the music running while switching, and to play shorter loops and switch so that identical sequence was heard on B and A.”

If it were the present crowd performing blind testing of amps, what would have been the possibility of them catching this issue let alone getting Bryston to fix it? They all believe there is no difference between amps in blind tests, right? But here we are.

What is the difference, these guys created a better and more revealing test. They test an amplifier's input against its (attenuated) output. That is what we are interested in, right? That is, an amp coloring its input or not. Testing random amp A against random amp B at heaven knows what volume and what content is not remotely as good as what they have done.

We're back to loudspeakers and preference testing again. No point in spending all those words. Loudspeakers don't even require blind testing in terms of identification. The audible differences aren't subtle enough that hearing bias even enters the equation. Fine for preference testing be we aren't and never have been talking about preference testing.
FMW is online now  
post #99 of 355 Old 02-04-2014, 09:10 AM
AVS Special Member
 
mcnarus's Avatar
 
Join Date: Nov 2007
Posts: 6,082
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 67 Post(s)
Liked: 293
What makes this whole debate silly is that even if every audio component DBT ever conducted were fatally flawed, it still wouldn't change the basic science, which shows that the output differences among various classes of audio equipment are too small to be heard by humans.

If you can't explain how it works, you can't say it doesn't.—The High-End Creed

mcnarus is offline  
post #100 of 355 Old 02-04-2014, 09:27 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by BruceD View Post

You miss the point entirely, sighted tests have no validity whatsoever as tests for audio comparisons, they are simply an exercise in displaying someone's personal agenda, especially people like you who have a vested interest in selling high dollar audio equipment.
That is your layman opinion. I shared what happens in real life, in the industry where we rely on sighted tests all the time. We do so in the interest of time and efficiency as I mentioned. As I explained, it is not a matter of if the test is sighted or not, it is a matter of whether the data is trustworthy or not. I take the sighted results of an expert listener over 100 layman blind test any day of the week and twice on Sunday smile.gif. Without any real experience of developing audio technology I appreciate your point of view which is limited to random joe reporting some sighted results. But that is not real world. That is just what happens in forums. Show me where real medical advances and discussions happen in forums vs outside. I don't think you will find any.

What agenda do you think an expert listener who is hired to better the development of a product has? His job is to find problems to resolve.

It is this myopic view of the science of audio evaluation that needs to be dispensed with. Until you have real life experiences, your generalizations simply do not apply. Have you ever been in a double blind test where you voted there was no difference but the other four did and turns out they were right? Sad to say, I have smile.gif. It is these sobering moments that gives one a balanced view. You have not experienced a double blind test until it matters. That is, your next mortgage payment depends on it. Then it is not a game of arguing on a forum. You want to take 5 years to develop a product or 5 months? If it is the latter, then you need to use all the tools at your disposal including proper sighted tests.

So no I didn't miss the point. I speak from experience. You speak from emotions related to winning a verbal cat fight on a forum. Your motivations are not mine. I am here to share how the science and engineering works in real life. If that is like learning how the sausage is made and hence distasteful, I can't help it smile.gif.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #101 of 355 Old 02-04-2014, 09:28 AM
Advanced Member
 
andyc56's Avatar
 
Join Date: Feb 2013
Posts: 575
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 17 Post(s)
Liked: 182
Quote:
Originally Posted by BruceD View Post

Quote:
Originally Posted by amirm View Post

. . . . . . I am a super strong supporter of the value of double blind tests. But I judge them as well as I judge sighted tests. . . . ;.

You miss the point entirely, sighted tests have no validity whatsoever as tests for audio comparisons, they are simply an exercise in displaying someone's personal agenda, especially people like you who have a vested interest in selling high dollar audio equipment.

The idea that both methods of evaluation should be criticized equally is a case of the balance fallacy, described very well here. A brief summary:
Quote:
The balance fallacy, also known as false balance, occurs when two sides of an argument are assumed to have equal value regardless of their respective merits.

Trapping people in this fallacy is an effective rhetorical technique, as can be seen in some posts above (not BruceD's). People don't want to be perceived as unfair, and their good will can be exploited.
andyc56 is offline  
post #102 of 355 Old 02-04-2014, 09:32 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by FMW View Post

We're back to loudspeakers and preference testing again. No point in spending all those words. Loudspeakers don't even require blind testing in terms of identification. The audible differences aren't subtle enough that hearing bias even enters the equation. Fine for preference testing be we aren't and never have been talking about preference testing.
Looks like you didn't read the post at all. The how to listen tool from Harman simply applies EQ at different levels to a flat response. If an expert listener can outperform you by a mile, then he does have better ears, i.e. "golden ears" relative to yours. If you can't even tell what has happened in such a linear transformation, how can you state that you are just as good as others in the more challenging non-linear distortions?

The second example was all about amplifiers. No reference to speakers. Did you read it? Nope. As I said, you all want a certain answer. If that answer is not what you want to hear, then the fighting begins and real facts and science presented are dismissed out of hands. "It is all speaker testing and preferences." An AB of an amplifier testing its input vs output is about preferences? Since when?

Come back when you take these discussions seriously. Until then you are just raising the noise floor of the discussion without adding anything to the discussion.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #103 of 355 Old 02-04-2014, 09:35 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,275
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 674 Post(s)
Liked: 1142
Quote:
Originally Posted by R Swerdlow View Post

And finally, from myself wink.gif:

Blind testing of anything involving human perception does not guarantee a valid test, but sighted testing guarantees an invalid test.

I'm a little bit less dogmatic about blind testing than that.

There are IMO differences in audio that are strong enough, such as a 10 dB overall level difference, that we can count on accurate responses from listeners without a blind tests.

The audible degradation caused by 16kbps MP3 coding is another example of it.

The original kinds of sound quality degradation that spurred our development of ABX were subtle differences. Some pretty fantastic claims about slew-rate limiting were being made at the time.

I for one do a fair amount of mixing for live sound and recording. I don't think I have to justify making 1 dB level changes by doing a DBT on the spot. However as Ethan Winer points out, just about every mixer has spent time fine tuning an adjustment on the console only to find out that the knobs being tweaked actually changed nothing that was in the signal path. Been there, done that! ;-)

Obviously determing when a carefully done DBT is needed involves a bit of a judgement call and perhaps some professionalism.
arnyk is offline  
post #104 of 355 Old 02-04-2014, 09:40 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by FMW View Post

We're back to loudspeakers and preference testing again. No point in spending all those words. Loudspeakers don't even require blind testing in terms of identification. The audible differences aren't subtle enough that hearing bias even enters the equation.
.
Missed that point about bias not entering speaker evaluation. You have some data to back that? Because I have data that says otherwise. Here is the result of Harman evaluation of sighted vs blind testing of speakers: http://seanolive.blogspot.com/2009/04/dishonesty-of-sighted-audio-product.html

BlindVsSightedMeanLoudspeakerRatings.png

These results look identical do you? If so, you are in dire need of real education about what blind testing is about. You think someone looking at these two speakers can look past their visual differences?

x108P363BK-o_other.jpeg

vs.

802-Diamond_Piano-Black-Gloss_OFF.jpg

How about the price difference? You think that doesn't enter the equation?

Let me guess: you did not follow double blind methodology to select your speakers. Right? This is what Dr. Olive says about that: "The Dishonesty of Sighted Listening Tests" when it comes to speaker evaluation.

Keep arguing. We can make this hole as deep as you like it smile.gif.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #105 of 355 Old 02-04-2014, 09:41 AM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,279
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 24 Post(s)
Liked: 38
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by FMW View Post

We're back to loudspeakers and preference testing again. No point in spending all those words. Loudspeakers don't even require blind testing in terms of identification. The audible differences aren't subtle enough that hearing bias even enters the equation. Fine for preference testing be we aren't and never have been talking about preference testing.
Looks like you didn't read the post at all. The how to listen tool from Harman simply applies EQ at different levels to a flat response. If an expert listener can outperform you by a mile, then he does have better ears, i.e. "golden ears" relative to yours. If you can't even tell what has happened in such a linear transformation, how can you state that you are just as good as others in the more challenging non-linear distortions?

The second example was all about amplifiers. No reference to speakers. Did you read it? Nope. As I said, you all want a certain answer. If that answer is not what you want to hear, then the fighting begins and real facts and science presented are dismissed out of hands. "It is all speaker testing and preferences." An AB of an amplifier testing its input vs output is about preferences? Since when?

Come back when you take these discussions seriously. Until then you are just raising the noise floor of the discussion without adding anything to the discussion.
so the test (which I have worked with some) is a test of expert listeners and not a training tool to increase skills. I didn't know that. Seems counter to what Sean has said.
Randy Bessinger is offline  
post #106 of 355 Old 02-04-2014, 09:53 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by Randy Bessinger View Post

so the test (which I have worked with some) is a test of expert listeners and not a training tool to increase skills. I didn't know that. Seems counter to what Sean has said.
It is both. When you guess incorrectly, that adds to your knowledge. If you keep running the test, over time, you get better at it as I shared. The tool scores how high a level you can achieve.

What I have said is completely consistent with Sean. Here he is from his subforum on WBF providing the link to the tool for the people to download and try and characterization of it: http://www.whatsbestforum.com/showthread.php?2984-Harman-s-How-to-Listen-Software-is-Now-Available-For-Download

"Well, it's been some time coming, but the listener training software Harman How to Listen is finally available for free download here...We hope you try the software, and find that it improves your critical listening skills."

So clearly it is educational. I don't recall if I asked Sean on WBF Forum or in person but he said their expert listeners must achieve level 12 before they can take on that job. That makes it a screening tool.

As an aside, Sean is now president of Audio Engineering Society!

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #107 of 355 Old 02-04-2014, 10:03 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by Randy Bessinger View Post

you are being mean spirited. Would you post that on your own forum.rolleyes.gif
Mean spirited about what? Talking about blind listening tests and B&W doing poorly? Of course I have talked about that in WBF Forum to chagrin of many members who fought me tooth and nail on them. Here is one of many threads: http://www.whatsbestforum.com/showthread.php?6491-It%92s-All-a-Preference. Go to post 23 among other.

I swear if you guys substitute 10% of your emotion for real research before jumping the gun like this, we would be in a far better place. It is just a hobby. No need to get angry. Next time you hit the quote button, think of how much technical knowledge you are sharing in that post. If it is zero, then perhaps you should not click....

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #108 of 355 Old 02-04-2014, 10:16 AM
AVS Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,279
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 24 Post(s)
Liked: 38
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by Randy Bessinger View Post

you are being mean spirited. Would you post that on your own forum.rolleyes.gif
Mean spirited about what? Talking about blind listening tests and B&W doing poorly? Of course I have talked about that in WBF Forum to chagrin of many members who fought me tooth and nail on them. Here is one of many threads: http://www.whatsbestforum.com/showthread.php?6491-It%92s-All-a-Preference. Go to post 23 among other.

I swear if you guys substitute 10% of your emotion for real research before jumping the gun like this, we would be in a far better place. It is just a hobby. No need to get angry. Next time you hit the quote button, think of how much technical knowledge you are sharing in that post. If it is zero, then perhaps you should not click....
Actually, I don't click that much. I have 1100 posts and you have 16000 (membership dates noted). I will just read for awhile now and take my emotions out on the winter weather here in kc. Have a good day:)
Randy Bessinger is offline  
post #109 of 355 Old 02-04-2014, 10:49 AM
AVS Special Member
 
Jack D Ripper's Avatar
 
Join Date: Jan 2014
Location: Burpelson Air Force Base
Posts: 1,136
Mentioned: 0 Post(s)
Tagged: 1 Thread(s)
Quoted: 4 Post(s)
Liked: 257
Quote:
Originally Posted by arnyk View Post
 
...

Stereo and multichannel as they presently are implemented are illusions. ...

 

Yes, indeed.  Anyone who listens to a two channel system should know that all of the sound is coming from two places, and yet if set up right, it subjectively sounds like there is sound coming from in between them as well (and often, audiophiles talk about the sound coming from wider than the placement of the speakers, and above them...).  If one had magic superhuman hearing, a stereo system would be useless for listening to music, as it is not the same as live at all.  So we know that human hearing is very flawed, and not the magic test equipment that so many golden ears pretend they are.  Human hearing is very limited, and very flawed.


God willing, we will prevail in peace and freedom from fear and in true health through the purity and essence of our natural fluids. God bless you all.
Jack D Ripper is offline  
post #110 of 355 Old 02-04-2014, 10:57 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by andyc56 View Post

The idea that both methods of evaluation should be criticized equally is a case of the balance fallacy, described very well here. A brief summary:
Trapping people in this fallacy is an effective rhetorical technique, as can be seen in some posts above (not BruceD's). People don't want to be perceived as unfair, and their good will can be exploited.

You agree with the opening quote then? "There's a kind of notion that everyone's opinion is equally valid. My arse! A bloke who's been a professor of dentistry for 40 years doesn't have a debate with some idjit who removes his teeth with string and a door! —Dara Ó Briain"

Maybe I should stop debating everyone as Dara says wink.gifsmile.gif.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #111 of 355 Old 02-04-2014, 10:58 AM
Senior Member
 
R Swerdlow's Avatar
 
Join Date: Nov 2007
Location: Gaithersburg, MD
Posts: 371
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 2 Post(s)
Liked: 32
Quote:
Originally Posted by Jack D Ripper View Post

… So we know that human hearing is very flawed, and not the magic test equipment that so many golden ears pretend they are.  Human hearing is very limited, and very flawed.

I don't worry so much about those ears, its what happens between the ears that seems to make for all the problems.
R Swerdlow is offline  
post #112 of 355 Old 02-04-2014, 11:07 AM
AVS Special Member
 
Ethan Winer's Avatar
 
Join Date: Apr 2003
Location: New Milford, CT, USA
Posts: 5,748
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 7 Post(s)
Liked: 133
Quote:
Originally Posted by jim19611961 View Post

For me, this muddies the results of blind A/B testing because we would have to know how trained the listeners are. In one test, you could have a room full of untrained listeners that would skew the results in the direction of there being no differences (or smaller ones) between A & B. And for all practical purposes, I know of no way to evaluate or test listeners to determine how acute their listening discernment really is.

Again, this is not a failing of blind tests, but of "public" tests generally. I'm sure a screening process could be devised using varying controlled amounts of distortion and lossy compression etc to evaluate potential participants. But for me this sums it up perfectly:

Forget public tests, and forget trained versus unskilled listeners. The people I want to blind test - and have asked to get together with many times - are those who themselves claim to hear things like lack of dither, or claim to be able to identify the "narrowing" of image width by jitter. I have literally begged some of these people to get together in person so I can (single) blind test them, and every one of them has refused.

--Ethan
Randy Bessinger likes this.

RealTraps - The acoustic treatment experts
Ethan's Audio Expert book

Ethan Winer is offline  
post #113 of 355 Old 02-04-2014, 11:26 AM
AVS Special Member
 
Jack D Ripper's Avatar
 
Join Date: Jan 2014
Location: Burpelson Air Force Base
Posts: 1,136
Mentioned: 0 Post(s)
Tagged: 1 Thread(s)
Quoted: 4 Post(s)
Liked: 257
Quote:
Originally Posted by FMW View Post

As a former high end audiophile I'll tell you what confuses me. All the decades I was involved in high end audio, my goal was to improve the sonics of my system in order to get as close to real as possible - or at least as close as I could afford to get. I was always interested in anything that dealt with the quality of sound reproduction. When I learned about bias controlled testing, I naturally took to it like a moth to a flame. What I learned from the experience changed the way I do audio forever. I wasn't angry that my $3500 CD player sounded exactly like my $100 one. I was thrilled that I could sell the expensive one and use the money for something more meaningful. Personally, I had no interest in bragging rights or impressing others. I was only interested in listening to music as accurately as I could. Removing an expensive piece of audio jewelry without losing anything sonic was terrific in my view.

Here is my confusion. Most audiophiles hate the concept that some of their audio jewelry doesn't necessarily represent sonic improvements. Why? A watch collector doesn't care that his Patek Phillippe watch doesn't keep better time than a Casio. The exotic car owner understands that a Ford will get him to destination just as effectively as a Lamborghini. Why do the audiophiles cling desperately to their sonic beliefs? Personally I think it is the concept of golden ears. I think audiophiles believe they can hear better than other people and therefore have a feeling of superiority over others. It isn't about equipment at all, perhaps. It is about their own skill level. They are furious that their hearing is biased just like everyone else's. It degrades that feeling of superiority.

I'm confused by that sentiment if it is true. Since high end audiophiles are generally higher up the intelligence and income scale, one would think they would look at things with a more analytical than emotional approach. The bias controlled listening test is an ear opener that every audiophile should experience. It is such a shame that it is such a fussy and difficult thing to do. I see no problem with wanting to own a $3500 CD player any more than owning a Patek Phillippe watch. It is a lovely thing to own. Why does it have to sound better?

 

I think you are right, that the situation with audio is different from those other luxury items, and I think you are right that it is that many audiophiles feel superior to others, in believing that they have some superhuman hearing rather than just the hearing that humans have.  It is also curious how much they often feel that makes them superior to others, as if super-hearing were a sign of intelligence rather than just a matter of hearing.  They do not seem to attribute such importance to animals that actually have superior hearing to humans.  Such an inconsistency runs counter to your assessment of their intelligence in your final paragraph.

 

I disagree with your concluding paragraph.  Although to have a lot of really expensive gear, one must have some money, I have not noticed any correlation between an interest in audio and intelligence.  For the most part, people with money are born with more than people who are quite poor.  Although people like Donald Trump like to pretend that they are financial geniuses who earned their riches, he was born rich.  He has made quite a few financial blunders, but when one is rich, one can afford to make mistakes that would devastate someone who is poor.

 

Also, the skills necessary for generating wealth are not at all the same skills as are required for being a good scientist.  So the fact that someone is able to generate wealth does not suggest that the person is particularly scientifically adept.


God willing, we will prevail in peace and freedom from fear and in true health through the purity and essence of our natural fluids. God bless you all.
Jack D Ripper is offline  
post #114 of 355 Old 02-04-2014, 11:29 AM
AVS Special Member
 
Jack D Ripper's Avatar
 
Join Date: Jan 2014
Location: Burpelson Air Force Base
Posts: 1,136
Mentioned: 0 Post(s)
Tagged: 1 Thread(s)
Quoted: 4 Post(s)
Liked: 257
Quote:
Originally Posted by Ethan Winer View Post
 
Quote:
Originally Posted by jim19611961 View Post

For me, this muddies the results of blind A/B testing because we would have to know how trained the listeners are. In one test, you could have a room full of untrained listeners that would skew the results in the direction of there being no differences (or smaller ones) between A & B. And for all practical purposes, I know of no way to evaluate or test listeners to determine how acute their listening discernment really is.

Again, this is not a failing of blind tests, but of "public" tests generally. I'm sure a screening process could be devised using varying controlled amounts of distortion and lossy compression etc to evaluate potential participants. But for me this sums it up perfectly:

Forget public tests, and forget trained versus unskilled listeners. The people I want to blind test - and have asked to get together with many times - are those who themselves claim to hear things like lack of dither, or claim to be able to identify the "narrowing" of image width by jitter. I have literally begged some of these people to get together in person so I can (single) blind test them, and every one of them has refused.

--Ethan

 

I think that that suggests, on some level, they know that they are full of hot air.  If someone really believes that they are superior, and has been bragging to others about it, one will typically love the opportunity to prove it if the superiority is real.  It is when it is BS that they do not want to be tested, as then they will be exposed for what they truly are.


God willing, we will prevail in peace and freedom from fear and in true health through the purity and essence of our natural fluids. God bless you all.
Jack D Ripper is offline  
post #115 of 355 Old 02-04-2014, 11:48 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,013
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 687 Post(s)
Liked: 388
Quote:
Originally Posted by Jack D Ripper View Post

I think that they suggests, on some level, they know that they are full of hot air.
No, they know that they will fail the test but they do not associate any value to that.
Quote:
If someone really believes that they are superior, and has been bragging to others about it, one will typically love the opportunity to prove it if the superiority is real.  It is when it is BS that they do not want to be tested, as then they will be exposed for what they truly are.
I have accepted such challenges in the past and in one case which happen to have been run here, found a major flaw in the test itself! Here is the thread on that: http://www.avsforum.com/t/908161/audio-dbt-1-summary. Note the section in red indicating how the author of the test copied duplicate files by accident. I found out two files were the same even though the author insisted that was not the case initially.

As you see in that thread, many people voted incorrectly and did not catch this mistake. Still want to insist that critical listening ability does not exist? And that double blind tests bring with them the aura of higher reliability? How often have you thought about the fact that mistakes could have occurred in how the test mechanics were conducted?

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #116 of 355 Old 02-04-2014, 12:11 PM
AVS Addicted Member
 
Chu Gai's Avatar
 
Join Date: Sep 2002
Location: NYC area
Posts: 14,753
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 146 Post(s)
Liked: 440
Quote:
Originally Posted by Randy Bessinger View Post


So clearly it is educational. I don't recall if I asked Sean on WBF Forum or in person but he said their expert listeners must achieve level 12 before they can take on that job. That makes it a screening tool.
Is this like Scientology with their levels?
Quote:
As an aside, Sean is now president of Audio Engineering Society! I follow him on twitter. He is heavily into headphones now, but not above marketing. I bought the jbl 700 headphones because of his posts. Disappointed for the money.
well, I hope you returned them! Gene over at AH has stated that in the AES meetings he's been to where Sean is press sent, that he never misses an opportunity to evangelize and market. Amazing though how Beats has just cornered the market.
Quote:
By the way, why is there hardly any participation in your experts threads? Sea, welti, seaton, etc haven't posted in a long time.
Anonymous sources have indicated a number of reasons.

Burnout
A significant number are on Time Out
Another group is on Double Secret Probation
Some of the industry professionals have determined their active participation does nothing to further their businesses so they have redirected their efforts elsewhere
How many times can you say "I really like my whatever" before you go insane
Incarceration
Other interests

Quite the pity seeing how it was AVS members who chased the true audiophiles out. Now some are homeless, nomads if you will.
Randy Bessinger likes this.

"I've found that when you want to know the truth about someone that someone is probably the last person you should ask." - Gregory House
Chu Gai is online now  
post #117 of 355 Old 02-04-2014, 12:35 PM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,328
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 83 Post(s)
Liked: 80
Quote:
Originally Posted by Ethan Winer View Post

Again, this is not a failing of blind tests, but of "public" tests generally. I'm sure a screening process could be devised using varying controlled amounts of distortion and lossy compression etc to evaluate potential participants. But for me this sums it up perfectly:

Forget public tests, and forget trained versus unskilled listeners. The people I want to blind test - and have asked to get together with many times - are those who themselves claim to hear things like lack of dither, or claim to be able to identify the "narrowing" of image width by jitter. I have literally begged some of these people to get together in person so I can (single) blind test them, and every one of them has refused.

--Ethan

Perhaps you would have more luck if you offered to do the testing at their place, rather than at yours.

While some claims are more plausible than others, there are those that take any claim that isn't a scientific one backed by scientific data to be Mr. Audiophile Golden Ears at it again rolleyes.gif

My Room
http://www.gearslutz.com/board/studio-building-acoustics/817205-my-listening-room.html

My Music
http://rateyourmusic.com/~jim1961

My Equipment

Rega - Apollo
Rega - DAC
Goldpoint Passive
(2) Classe CA-100 bridged power amps (350w)
Jenzen Next ( http://www.troelsgravesen.dk/Jenzen-NEXT.htm )
...
jim19611961 is online now  
post #118 of 355 Old 02-04-2014, 01:02 PM
AVS Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,276
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 113 Post(s)
Liked: 173
Quote:
Originally Posted by nvidio View Post

Yes, that is entirely correct. However, if it leads them to conclude that 'something must be wrong with blind testing' then it is up to you to, instead of repeating a million times over that nothing can possibly be wrong with blind testing, prove that nothing is wrong with blind testing. Wanna know why I want you to prove it? It's actually quite easy to understand. Just to make it even more easy, let me give you a small hint. It's because, despite the fact this has already been made perfecly clear to you from the very beginning, you are still consistently failing to admit that the real reason why you are still consistently failing to prove it is only because you can't. rolleyes.gif

Those who say 'something must be wrong' need to be a bit more specific. *What* must be wrong? If they challenge a specific aspect of a blind testing protocol, that can be tested. A hysterical faith in the infallibility of one's own senses is not an argument.

And again, you have been pointed to scientific literature on sensory testing. Why do you refuse to acknowledge it?


You're like the anti-evolutions camp who insist they have never been shown a *real* example of a 'transitional fossil'....a 'proof' they insist on.
krabapple is offline  
post #119 of 355 Old 02-04-2014, 01:18 PM
AVS Addicted Member
 
Chu Gai's Avatar
 
Join Date: Sep 2002
Location: NYC area
Posts: 14,753
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 146 Post(s)
Liked: 440
You need training, Krab! Army training.

hqdefault.jpg

"I've found that when you want to know the truth about someone that someone is probably the last person you should ask." - Gregory House
Chu Gai is online now  
post #120 of 355 Old 02-04-2014, 01:18 PM
Member
 
nvidio's Avatar
 
Join Date: Nov 2010
Location: beer city (aka Belgium)
Posts: 188
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 20
Quote:
Originally Posted by arnyk View Post

Using audible illusions to discredit audio DBTs is like using tax cheaters to discredit the idea of taxation.

If we all sat around doing nothing listening to Poppy Crumb Audible Illusions recordings on our audio systems...

LOL!
Last time I checked, music was the art of creating audible illusions.
nvidio is offline  
Closed Thread Audio theory, Setup and Chat

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off