Why Blind Listening Tests Are Flawed? Per Robert Harley! - Page 7 - AVS Forum
Forum Jump: 
Reply
 
Thread Tools
post #181 of 914 Old 06-20-2009, 01:27 PM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,114
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 142 Post(s)
Liked: 384
Quote:
Originally Posted by amirm View Post

So how do we get the other answers?

I think you're getting off topic here. We're addressing the efficacy of blind vs. non blind listening tests in answering the question of whether there's an audible difference. Other audio questions are a topic for a different thread.


Quote:


Again, you are misreading the experiment. I asked if you think you may have noticed a difference, which way you vote. Your taste buds are not binary devices. Neither is your hearing. When faced with shades of gray, what do you do? What if it is a shade of gray for you, but absolute for me. Should we average the two results?

I shouldn't have said I would vote "I don't know", since the ABX test means you MUST choose whether X is A or B. I would make a note that I was just guessing, though. I think you misread the purpose of an ABX test. It is NOT a qualitative evaluation. It simply asks which is which (there's no "shades of gray" involved). If the difference is audible, this is straightforward. If it isn't audible, then the test will show that you're just guessing.

Quote:


People on both sides belittle measurements when it fits their argument. In our long running DAC test thread (which was a blind test that I ran by the way), objectivists would refuse to accept better measurement of the winning product as having any value. Their claim is that some measurements are too precise to matter. Which may be true but that admission clearly disputes your generalization that only subjectivists dismiss measurements.

I didn't make myself sufficiently clear. Subjectivists claim they hear things that can't be measured (and no, I am NOT talking about one's emotional response to music. I am talking about SOUND). Objectivists state that some things can be measured but not heard (THAT's what subjectivists belittle). If one hears something, it's because of

a) A biased perception, or
b) A corresponding measurement

The blind test allows us to eliminate a, so we can focus on b. That's what subjectivists don't like.

Quote:


But let’s say are right. Tomorrow I want to go and buy an amp. Where would I find the double-blind tests of the two amps I see in the store?

If you're lucky, you can find the results on-line. There have also been RARE magazine articles about this. Unfortunately, you'll sure as hell NEVER see such a test in a High End audio mag.
RobertR is online now  
Sponsored Links
Advertisement
 
post #182 of 914 Old 06-20-2009, 02:17 PM
 
ChrisWiggles's Avatar
 
Join Date: Nov 2002
Location: Seattle
Posts: 20,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 14
Quote:
Originally Posted by RobertR View Post

I've never seen anyone claim that it provides all answers to all questions. What it DOES answer is "are there differences once one removes the source(s) of bias". That's what we're discussing.

No. As I keep attempting to explain, that conclusion cannot be drawn from that test. You can only address whether there are differences that are audible within the limitations and threshold of THAT test. And all it says is that given one particular test methodology, there were no audible differences that were found for that test. It doesn't say there were no audible differences at all that might be found with another methodology, just that none could be found using that method.
ChrisWiggles is offline  
post #183 of 914 Old 06-20-2009, 02:20 PM
AVS Addicted Member
 
Alimentall's Avatar
 
Join Date: Aug 2004
Location: Home by the sea
Posts: 14,157
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
And with the limits of existing speaker technology. Listening for amplifier/cable/source differences with existing speaker technology is like searching for a needle in a haystack.

John
Alimentall is offline  
post #184 of 914 Old 06-20-2009, 02:26 PM
 
ChrisWiggles's Avatar
 
Join Date: Nov 2002
Location: Seattle
Posts: 20,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 14
Quote:
Originally Posted by Raul GS View Post

I did get it, I just didn't explain myself well enough. If you consider your second point quoted, in essence that is an argument that relies on "you can't prove a negative" to make its claim. IOW, even though your claim is correct, test methodology can limit hearing threshold, your statement that "another test might discern" needs to be established, not assumed. That is, you cannot expect one to prove the negative. The individual making the claim has to prove it in a manner that is objective and repeatable. Otherwise we are left in a realm where no claims can EVER be made, because there is just no test to figure it out yet...does that sound familiar?

Of course claims can be made. If one claims there is a difference, and blind testing finds a difference, there is a difference. Simple as that.

If someone claims there is a difference, and some blind test finds no difference, we don't really know whether there is any difference, we only know that there wasn't a difference significant enough to be audible in that test. In this situation we have still learned something about the significance (in this case minimal) of any possible difference that is being claimed. If the cost difference between two components is large, and it's not an obvious enough difference to hear in a particular test method, then I might say "well gee, there might possibly be some difference yet to be discerned in a blinded test, but is such a tiny possible difference even worth my money?" We can also draw from other information about what difference is being claimed. If the claim is about something that we can objectively know to be extremely unlikely (cable elevators, power cord tweaks, etc) then I might draw an informed conclusion from information outside of listening tests that there is not likely to be any difference at all. From this I would proceed with an informed assumption that there are no differences unless evidence to the contrary arises.

Where the onus lies depends on what's being tested. If, for instance, someone is claiming all amplifiers sound the same, or all speakers sound the same, the onus lies with that claim, IMO. If someone is claiming that speaker cables of equal gauge can still make a difference depending on some other pseudo-science factor, then the onus is on them. Where the onus is doesn't matter to the conclusions we can draw from a blind test. Just as Harley is wrong in disregarding testing that doesn't agree with his non-objective beliefs, so too are people wrong in drawing erroneous or over-reaching conclusions from a particular blind test.
ChrisWiggles is offline  
post #185 of 914 Old 06-20-2009, 02:26 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by Alimentall View Post

And with the limits of existing speaker technology. Listening for amplifier/cable/source differences with existing speaker technology is like searching for a needle in a haystack.

I use high quality headphones and it in no way is a limitting factor in detecting such differences.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #186 of 914 Old 06-20-2009, 02:51 PM
QQQ
AVS Addicted Member
 
QQQ's Avatar
 
Join Date: Jan 2002
Posts: 10,998
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 13
Quote:
Originally Posted by RobertR View Post

I've never seen anyone claim that it provides all answers to all questions. What it DOES answer is "are there differences once one removes the source(s) of bias". That's what we're discussing.

Quote:
Originally Posted by ChrisWiggles View Post

No. As I keep attempting to explain, that conclusion cannot be drawn from that test. You can only address whether there are differences that are audible within the limitations and threshold of THAT test. And all it says is that given one particular test methodology, there were no audible differences that were found for that test. It doesn't say there were no audible differences at all that might be found with another methodology, just that none could be found using that method.

I think you are arguing semantics somewhat to the extreme here Chris. I think it's pretty well established that you can't prove a negative and the intent of what Robert said was quite clear. He was not arguing what you are suggesting he was.
QQQ is offline  
post #187 of 914 Old 06-20-2009, 04:41 PM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,114
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 142 Post(s)
Liked: 384
Quote:
Originally Posted by QQQ View Post

I think you are arguing semantics somewhat to the extreme here Chris. I think it's pretty well established that you can't prove a negative and the intent of what Robert said was quite clear. He was not arguing what you are suggesting he was.

You got it, thanks.
RobertR is online now  
post #188 of 914 Old 06-20-2009, 07:20 PM
AVS Special Member
 
Raul GS's Avatar
 
Join Date: Feb 2003
Posts: 2,193
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 24
Quote:
Originally Posted by ChrisWiggles View Post

If someone claims there is a difference, and some blind test finds no difference, we don't really know whether there is any difference, we only know that there wasn't a difference significant enough to be audible in that test... If the claim is about something that we can objectively know to be extremely unlikely (cable elevators, power cord tweaks, etc) then I might draw an informed conclusion from information outside of listening tests that there is not likely to be any difference at all.

The dividing line you are suggesting is not as clear as you seem to believe. The "objectively know" line is exactly what is in question from the subjective perspective, and the rules as set out in your first statement validates it. That is why people making the claim must prove their claim, otherwise you are requiring people to prove a negative. That is, anything goes, because the subjectivist can always rely on the argument that "we only know that there wasn't a difference significant enough to be audible in that test"

That being said, tests may be poorly designed and hide small differences, but that is why people are required to describe the testing protocol, to determine if the process was conducive to produce false positives, or in this case false negatives.

Ignorance more frequently begets confidence, than it does knowledge. Charles Darwin
Raul GS is offline  
post #189 of 914 Old 06-20-2009, 09:24 PM
 
ChrisWiggles's Avatar
 
Join Date: Nov 2002
Location: Seattle
Posts: 20,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 14
Quote:
Originally Posted by Raul GS View Post

The dividing line you are suggesting is not as clear as you seem to believe. The "objectively know" line is exactly what is in question from the subjective perspective, and the rules as set out in your first statement validates it. That is why people making the claim must prove their claim, otherwise you are requiring people to prove a negative. That is, anything goes, because the subjectivist can always rely on the argument that "we only know that there wasn't a difference significant enough to be audible in that test"

That being said, tests may be poorly designed and hide small differences, but that is why people are required to describe the testing protocol, to determine if the process was conducive to produce false positives, or in this case false negatives.

I agree. All I wanted to say is that a negative result on a blind test is not proof of the absence of all possible differences.

But generally speaking, it's fair to place the burden of proof on those claiming audible(or visual) differences, and I agree with you there.

Again, Q is probably right, I'm being semantic about it, but precision here is important when you're trying to draw conclusions from testing. One has to be careful to reach only that conclusion which is supported by the data, and not exceed that.

It may be absolutely true that the two amps above are not at all audibly different, but that's not a conclusion that can be drawn and supported from a single test.

And yes, this does give subjectivists something of an out. That doesn't mean they're right at all, it just means that they're stubborn and aren't swayed by the preponderance of the data. I have a problem with that.
ChrisWiggles is offline  
post #190 of 914 Old 06-20-2009, 09:26 PM
 
ChrisWiggles's Avatar
 
Join Date: Nov 2002
Location: Seattle
Posts: 20,730
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 14
Quote:
Originally Posted by QQQ View Post

I think you are arguing semantics somewhat to the extreme here Chris. I think it's pretty well established that you can't prove a negative and the intent of what Robert said was quite clear. He was not arguing what you are suggesting he was.

So you're suggesting that I was arguing with what you thought I thought his intent was in his argument, which was not what I suggested it was he was arguing...?

I'm not that smart Q. I'm lost. What was my position again?
ChrisWiggles is offline  
post #191 of 914 Old 06-21-2009, 04:47 AM
Senior Member
 
hevi's Avatar
 
Join Date: Jan 2007
Posts: 365
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 33 Post(s)
Liked: 48
Quote:
Originally Posted by amirm View Post

The Swedish LTS claims all amps but one color the sound.

No. They do not. Their records, however, show that using their test methodology, they have been able to proove audible coloration with statistical significance in all *tested* amps but a couple, of wich only one is still in production (they've used this test methodology for 20 odd years).

Quote:
Originally Posted by amirm View Post

Yet that directly contradicts the views of many objectivists who will say that in a double-blind tests, no one can tell the difference between "properly designed" amps.

Without stating the test setup such a claim is meaningless.

To put LTS results in perspectice, the test setup LTS use to test audible artifacts from an amplifier is far more sofisticated and sensitive than a simple A/B test setup. With LTSs test setup you have the means to vary the amp load and listening level independantly from eachother, making it immensly more easy to hear any colorations from the amp, when it is taken all the way from idling up to its very limit whilst driving an artificial (but very realistic) "semi-difficult" loudspeaker load.

H

hevi is offline  
post #192 of 914 Old 06-21-2009, 05:50 AM
AVS Special Member
 
clehner's Avatar
 
Join Date: Feb 2000
Location: Berlin, Germany
Posts: 1,214
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I think the discussion is totally futile.

Subjectivist and objectivist views are incommensurable. However, a subjectivist can profit from objectivist reasoning.

Let me briefly explain what I mean. I personally am convinced that I will not be able to distinguish between two high end amplifiers (but I always can hear the PAL speedup, no matter what, same with two different projectors, and I will be happy to do a test if somebody wants to organize one ). I rather doubt that I will be able to tell the difference between my not so expensive consumer receiver vs a 10 times expensive piece (under scientific testing conditions).

Now, if any subjectivist claims his/her amplifier is vastly superior to another piece of equimpent, let him/her prove it in a DBX. Some subjectivists will even find it rather intriguing to put themselves to a test like that as they might be able to proof something there (that their hearing is superior to average hearing or the like).

However, as has been pointed out before, hard core subjectivsts will never accept that and claim all testing (maybe even science itself) is flawed. Well, that marks the border between religion and science, so there is no way to reach any result no matter how long you will keep up the discussion.

I am not judging here, don't get me wrong. All I say is, there is no way of bringing both world views together.

Regards
Christoph
*************
Contact via Public Profile
clehner is offline  
post #193 of 914 Old 06-21-2009, 06:29 AM
AVS Club Gold
 
Art Sonneborn's Avatar
 
Join Date: Jun 2000
Location: Battle Creek,MI USA
Posts: 22,302
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 44
Quote:
Originally Posted by clehner View Post

However, as has been pointed out before, hard core subjectivsts will never accept that and claim all testing (maybe even science itself) is flawed. Well, that marks the border between religion and science, so there is no way to reach any result no matter how long you will keep up the discussion.

I am not judging here, don't get me wrong. All I say is, there is no way of bringing both world views together.

My point before. So much of what the subjectivist claim is dismissing science but accepting the differences noted etc almost on faith. In fact the subjectivist arguments are so akin to religion this whole discussion is wasted IMO. Even if the obective position can show scientifically that most if not all of the subjectivist positions are just built from the "it must be believed to be seen" realm this would never be sufficient to convince them.

What gets me most is there are guys here who are scientists and they would have no problem applying science rigorously to other subjects but for some reason give this one a pass. Of cousew I personally will do that with my religous beliefs but that simply illustrates my point here.

Art

My HT


iRule rules my theater
 

"If she's amazing she won't be easy,if she's easy she won't be amazing"

 

Bob Marley

Art Sonneborn is offline  
post #194 of 914 Old 06-21-2009, 07:29 AM
AVS Addicted Member
 
Alimentall's Avatar
 
Join Date: Aug 2004
Location: Home by the sea
Posts: 14,157
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
I find that real differences become much more dramatic when you can instantly switch from one product to another at the same volume, while imaginary differences vanish pretty quickly. The problem here is that most magazines and audiophile systems rely on gigantic imaginary differences for their 'enjoyment' and don't want to see that showed to be little or nothing in a DBT. As in the time one prospect said "I can't possibly audition your speakers with that [$500 award-winning] NAD CD player". I said "Okay, why don't you listen to the speakers first, see how you like them here, then imagine them sounding even better with your CD player". Personally, I don't care that much, I've gotten so annoyed by audiophilia nervosa over the last 5 years or so that I gave up entirely on 'high-end' and just started worrying about 'high performance'. People not willing to audition things based on price or aesthetics or name brand are not 'audiophiles', they're just plain snobs.

John
Alimentall is offline  
post #195 of 914 Old 06-21-2009, 08:04 AM
AVS Special Member
 
clehner's Avatar
 
Join Date: Feb 2000
Location: Berlin, Germany
Posts: 1,214
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 0 Post(s)
Liked: 10
Quote:
Originally Posted by Art Sonneborn View Post

What gets me most is there are guys here who are scientists and they would have no problem applying science rigorously to other subjects

Yes, thanks Art, sorry I did not remember it was you ...

Albert Einstein e.g. was religious (in a general vague sense of religion), I don't think there is a problem here, as both worlds are just on different philosophical levels. You can't convince somebody with strong religious believes that something he believes in just isn't possible without abondoning the laws of physics.

@Alimentall:
Quote:


I've gotten so annoyed by audiophilia nervosa over the last 5 years or so that I gave up entirely on 'high-end' and just started worrying about 'high performance'.

Exactly my point, it's not as if there weren't lots of (theoretical and practical) problems still to be solved in home theater. No reason to chase after ghosts (if you are bored).

Regards
Christoph
*************
Contact via Public Profile
clehner is offline  
post #196 of 914 Old 06-21-2009, 08:21 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by hevi View Post

No. They do not. Their records, however, show that using their test methodology, they have been able to proove audible coloration with statistical significance in all *tested* amps but a couple, of wich only one is still in production (they've used this test methodology for 20 odd years).

You say "NO" but then go on and repeat exactly what I said! Of course it is "using their methodology." What else would it be? Something they dreamed in the middle of the night?
Quote:


To put LTS results in perspectice, the test setup LTS use to test audible artifacts from an amplifier is far more sofisticated and sensitive than a simple A/B test setup. With LTSs test setup you have the means to vary the amp load and listening level independantly from eachother, making it immensly more easy to hear any colorations from the amp, when it is taken all the way from idling up to its very limit whilst driving an artificial (but very realistic) "semi-difficult" loudspeaker load.

H

So what is it? Their test reflects real life or not? If it reflects real life, then I should be able to hear the same "coloration" by other means. If I cannot, then their tests are meaningless. I can't tell which point of view you are advocating.

If you are saying that in a normal A/B test I will not be able to repeat their results, then you are saying A/B tests are not a good way to find differneces in amps. Ohterwise, you are damning LTS methodology if the same results cannot be gained through A/B test. Please clarify.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #197 of 914 Old 06-21-2009, 08:40 AM
AVS Special Member
 
Raul GS's Avatar
 
Join Date: Feb 2003
Posts: 2,193
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 3 Post(s)
Liked: 24
Quote:
Originally Posted by hevi View Post

they have been able to proove audible coloration with statistical significance in all *tested* amps but a couple, of wich only one is still in production (they've used this test methodology for 20 odd years).

As you may have noticed, I do take Amir to task with some of his claims, but he is dead right on NIN and now with you. You both clearly state that the test proved most amps exhibit colorations. Who cares if it is minor, if it is under stress (especially since so many high-end speakers present difficult loads), blah, blah, blah. You both clearly acknowledge that all tested amps but 2 had colorations. Now lets consider Chris's point, that some tests are not sensitive enough to show small differences, and now we have Amir's conclusion, most of those prior amp tests (and by extension any electronics, even cable tests) failed to show a difference because the tests were faulty.

Now, you may argue that the amp differences are so small they don't reflect at all the claims made by "golden eared" reviewers and audiophiles. You may also claim that in most cases, factors such as room effects, speaker placement, and so on, would swamp the differences. That may be so, but the bottom line is, the test methodology commonly used by objectivists is now put into question with regards to tiny differences, and you and NIN are the culprit

Ignorance more frequently begets confidence, than it does knowledge. Charles Darwin
Raul GS is offline  
post #198 of 914 Old 06-21-2009, 08:42 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by Art Sonneborn View Post

My point before. So much of what the subjectivist claim is dismissing science but accepting the differences noted etc almost on faith.

Art, I hope you are not putting me in this bucket . I have practiced more of the science of double-blind testing than everyone in this thread or others like it. Indeed, when I ask who has participated in formal double-blind tests time and time again, no one raises their hands. When I suggest tests that people can run on their own to get a feeling for that, they do not jump on that either.

It is after practicing the science that one learns how little we still know about what is going on in audio. I wish we had all the answers. I wish we could measure "fidelity." I wish everyone in the world was a trained listener and knew what to look for when searching for artifacts like jitter. I wish there was a way to perform double-blind tests with zero delay in all cases as to not rely on poor memory of people. And I wish it was free to conduct double-blind tests and didn't take so much time to do them. Sadly, none of these wishes are true today. If there were, trust me, everyone would be thinking alike.

Yes, there are many people who are into hyperbole. Sadly this is true of both sides. People go read a THD spec for a mass market 5.1 amp selling for $400 and think that means the amp is as good as one from a company which cares about audio design and not just the dollar amount and stated wattage. And at the same time, there are subjectivists who buy into stuff that do not have science backing them.

Given the fact that both sides are guilty of some sins, it only makes sense for one to live in both world. Appreciate what each side has to say and use it for what it is worth. 30 years ago my brother who grew up with Tubes challenged me that he could build a tube amp that would sound better than my then monster stereo amp. I laughed at him. Two days later, he has a single tube amp ready for the challenge. He hooks the thing up to one speaker and we use my amp to drive the other speaker. You would not believe the shock in my face of how much more enjoyable his amp was. Of course, turn up the volume and my solid-state amp killed his. But at low volumes, I would much rather listen to his amp than mine. Of course, I did not let him know and for the next 10+ years, was a staunch measurement and objectivist guy. But over time, one goes through more experiences like this, allowing him to leave open the possibility that the other side may also be right in what sounds good to them.

Quote:


Even if the obective position can show scientifically that most if not all of the subjectivist positions are just built from the "it must be believed to be seen" realm this would never be sufficient to convince them.

This goes both ways Art. Consider that subjectivists have as much or even stronger “science” on their side. I can show you with real measurements that jitter exists, what its spectrum is, and magnitude. I can show you that the equipment that an audiophile likes measures much better than one he doesn’t. I can explain why jitter degrades sound, and through real math and proof, show that its level must be below what exists in mass market gear to reproduce 16-bits at full response. I can also show how jitter gets into the DAC clock. And what is the reaction of the objectivist? Well, none of that matters because jitter is not audible. They may be right but that is not the point. The point is that there is science that guides us in subjective differences. That we can’t conduct the right experiment here when jitter can have infinite variety is more of a problem than dismissing science.

Quote:


What gets me most is there are guys here who are scientists and they would have no problem applying science rigorously to other subjects but for some reason give this one a pass. Of cousew I personally will do that with my religous beliefs but that simply illustrates my point here.

Art

And that gets us to the best position to be: not having an extreme position on either side or by definition, you are practicing religion. Let the data from the other side to sink in. Then conduct your own experiments. Take what you know to be true, then subject yourself to double-blind tests and see if the results are satisfying. If they are, then you are good to go. But standing on one side of the fence and wondering why the other side is so dogmatic, is not going to be satisfying, as long as you choose to participate in threads like this .

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #199 of 914 Old 06-21-2009, 09:42 AM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,114
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 142 Post(s)
Liked: 384
Quote:
Originally Posted by amirm View Post

what is the reaction of the objectivist? Well, none of that matters because jitter is not audible. They may be right but that is not the point.

But it IS the point is if it's not audible below a certain level once bias is removed (that's the real proof, not "it's audible because my theory says it should be"). As I stated before, not all that can be measured can be heard.
RobertR is online now  
post #200 of 914 Old 06-21-2009, 10:29 AM
Senior Member
 
hevi's Avatar
 
Join Date: Jan 2007
Posts: 365
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 33 Post(s)
Liked: 48
Quote:
Originally Posted by amirm View Post

You say "NO" but then go on and repeat exactly what I said! Of course it is "using their methodology." What else would it be? Something they dreamed in the middle of the night?

You wrongly state in the original message that "The Swedish LTS claims all amps but one color the sound." I point out to you that they have never claimed that since your claim imply they have tested all amps in the world. Which they have not.

Quote:
Originally Posted by amirm View Post

So what is it? Their test reflects real life or not? If it reflects real life, then I should be able to hear the same "coloration" by other means. If I cannot, then their tests are meaningless. I can't tell which point of view you are advocating.

That's because I do not advocate any point. I'm clarifying the test setup they use and a plausible explanation as to how they can detect minute colorations even in well engineered amplifiers, where other test setups fail.

Quote:
Originally Posted by amirm View Post

If you are saying that in a normal A/B test I will not be able to repeat their results, then you are saying A/B tests are not a good way to find differneces in amps. Ohterwise, you are damning LTS methodology if the same results cannot be gained through A/B test. Please clarify.

Again, I am not saying anything about the vaildity of their tests, I am clarifying their test procedure.

If you want my personal oppinion, I find their test setup pretty appealing (regardles of it beeing done blind) since they have the ability to take an amp from a normal A/B setup and gradually take it to its limits when it comes to percieved neutrality. Some amps are found to color at low to normal output levels, others crumble early in the load cycle, and some stay neutral way up in power. If the amp is colored they usually provide under what conditions that happen and subjective information about the character of the coloration, so even if you're not looking for a neutral amp their tests can be very helpful in finding good candidates to audition.

H

hevi is offline  
post #201 of 914 Old 06-21-2009, 10:39 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by RobertR View Post

But it IS the point is if it's not audible below a certain level once bias is removed (that's the real proof, not "it's audible because my theory says it should be". As I stated before, not all that can be measured can be heard.

And how do you suggest we find that 'certain level?' Just declare it and be done with it? Where is the science in that?

Take jitter. It has level, spectrum and whether it correlates with the source, is random or periodic. That combination makes it impossible to come up with any kind of metric to say if it does or does not matter. Or any kind of test that would prove what is audible because you wouldn't know which set of parameters to test. Furthermore, we can prove using simple math, that you must have a jitter spec well below what most gear can muster, to resolve the 16-bits of audio samples you have purchased.

Calling such science 'theory,' is as harsh of a characterization to me as folks not believing in double-blind tests .

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #202 of 914 Old 06-21-2009, 11:04 AM
Senior Member
 
hevi's Avatar
 
Join Date: Jan 2007
Posts: 365
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 33 Post(s)
Liked: 48
Quote:
Originally Posted by Raul GS View Post

As you may have noticed, I do take Amir to task with some of his claims, but he is dead right on NIN and now with you. You both clearly state that the test proved most amps exhibit colorations. Who cares if it is minor, if it is under stress (especially since so many high-end speakers present difficult loads), blah, blah, blah. You both clearly acknowledge that all tested amps but 2 had colorations. Now lets consider Chris's point, that some tests are not sensitive enough to show small differences, and now we have Amir's conclusion, most of those prior amp tests (and by extension any electronics, even cable tests) failed to show a difference because the tests were faulty.

Not really. *I* do not state *anything* about the validity of LTS tests in the post you are refering to.

I point out that amirm wrongly state in the original message that "The Swedish LTS claims all amps but one color the sound." ,which they have *not*, folowed by a layman explanation about the test procedure LTS use.

However, as stated in another post, I admit I'm somewhat intrigued by their tesiting methodology.

H

PS. Just for the record, just because "NIN" and I are both Swedish and know about LTS dosn't automatically mean we share the same oppinions (or that I am a militant objectivist, which I am not).
DS.

hevi is offline  
post #203 of 914 Old 06-21-2009, 11:04 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by hevi View Post

You wrongly state in the original message that "The Swedish LTS claims all amps but one color the sound." I point out to you that they have never claimed that since your claim imply they have tested all amps in the world. Which they have not.

You really say with a straight face that I claimed they had tested every amp in the world? Of course that is not the case. They do however, talk as if no amp could be transparent but one: http://bryston.com/pdfs/07/Swedish14BSSTReview.pdf

"To sum it up, normally there are lots of views, ideas and opinions regarding the character of the tested amplifier after the open listening. That was not the case this time.

We were sitting in open listening for well over one hour, and no one mentioned a single word about any differences they either imagined or heard. Actually, that's the first time ever something like that has happened."


So when I read that that they are so surprised to find the first amp that sounds the same as the original, my conclusion is clear: they do consider amps to sound different based on 20 years of testing them. Yet there are few objectivists who accept that claim.

Quote:


That's because I do not advocate any point. I'm clarifying the test setup they use and a plausible explanation as to how they can detect minute colorations even in well engineered amplifiers, where other test setups fail.

We can all read their methodology: http://www.sonicdesign.se/amptest.htm. Thankfully it is in English and we can read it. The issue at hand then is what their results mean.
Quote:


If you want my personal oppinion, I find their test setup pretty appealing (regardles of it beeing done blind) since they have the ability to take an amp from a normal A/B setup and gradually take it to its limits when it comes to percieved neutrality.

I also consider their test clever. But have a problem with them violating test protocol by allowing testers to talk to each other and as such, introduce bias into the results:

"When the blind verifying listening thereafter takes place, it is still an option to talk to the other listeners, but of course it is difficult to draw any conclusions since it is no longer known which one is B or A."

They are mistaken why this does not matter. Yes, no bias is introduced in favor of one product or the other. But if one person's opinion is borderline, they can be pushed over the line one way or the other by other testers: "didn't you hear that? It sounds worse right there. Doesn't it?"

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #204 of 914 Old 06-21-2009, 11:38 AM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,114
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 142 Post(s)
Liked: 384
Quote:
Originally Posted by amirm View Post

And how do you suggest we find that 'certain level?'

I would have thought the answer was obvious, since the thread is about double blind testing....
RobertR is online now  
post #205 of 914 Old 06-21-2009, 12:02 PM
Senior Member
 
hevi's Avatar
 
Join Date: Jan 2007
Posts: 365
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 33 Post(s)
Liked: 48
Quote:
Originally Posted by amirm View Post

You really say with a straight face that I claimed they had tested every amp in the world?

No, I wasn't trying to put words in your mouth, I was simply trying to explain why I find your claim "The Swedish LTS claims all amps but one color the sound" to be incorrect (or incomplete, really). I figure since you're a contributing editor you should be good enough with words to make clear unambigous statements.

Quote:
Originally Posted by amirm View Post

So when I read that that they are so surprised to find the first amp that sounds the same as the original, my conclusion is clear: they do consider amps to sound different based on 20 years of testing them. Yet there are few objectivists who accept that claim.

Perhaps their test setup is just so extremely demanding and revealing that they did not expect an amp to pass wihout detecting any colorations?

Quote:
Originally Posted by amirm View Post

"When the blind verifying listening thereafter takes place, it is still an option to talk to the other listeners, but of course it is difficult to draw any conclusions since it is no longer known which one is B or A."

They are mistaken why this does not matter. Yes, no bias is introduced in favor of one product or the other. But if one person's opinion is borderline, they can be pushed over the line one way or the other by other testers: "didn't you hear that? It sounds worse right there. Doesn't it?"

I agree in a way, yes, but in another "not really". If the "sufficient statistical significance" is on a "per individual" basis you only need one person picking out the amp with statistical significance and you end up with this scenario:

If the amp is infact coloring the signal. Any other "followers" that just do what the "manipulative dude with the good hearing" say do not matter for the significance since you only need one in the test panel to pick it out, and you allready have that guy.

However, if the amp is instead percieved as "transparent" by all test persons, manipulation may be the reason, or perhaps their ears where all clogged up, or they where all hung over or what not. Anyway, since this has happened two times in the past 20 years they probably could do further testing with stricter rules when finding an amp that they can't beat into submission but instead they just decided to call it a day. After all, these people do this in their spare time and not to provide the world with inpeckable test results about transparent amplifiers.

H

hevi is offline  
post #206 of 914 Old 06-21-2009, 01:03 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by hevi View Post

Perhaps their test setup is just so extremely demanding and revealing that they did not expect an amp to pass wihout detecting any colorations?

Correct. Which leads one to believe that according to them, amps do color and are not transparent. Given that, I ask how many people who believe in double-blind testing agree that all the amps they tested in 20 years, lack transparency.

Quote:


I agree in a way, yes, but in another "not really". If the "sufficient statistical significance" is on a "per individual" basis you only need one person picking out the amp with statistical significance and you end up with this scenario:

If the amp is infact coloring the signal. Any other "followers" that just do what the "manipulative dude with the good hearing" say do not matter for the significance since you only need one in the test panel to pick it out, and you allready have that guy.

Ah, you are making an assumption which we do not know to be true. What if that vocal person is dead wrong? You know, like the people who think "cables sound different." And he manages to convince everyone else that way. Now we have a "blind test with population >> 1" claiming results that are only represenative of the one person who is in fact mistaken.

Bottom line is that we use multiple testers as to build up more confidence in the result, and lower the chance that a person voted mistakenly. By having people talk together, the testing validity reduces to much lower population. I would rank group bias almost as high as having the test non-blind.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #207 of 914 Old 06-21-2009, 02:13 PM
Senior Member
 
hevi's Avatar
 
Join Date: Jan 2007
Posts: 365
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 33 Post(s)
Liked: 48
Quote:
Originally Posted by amirm View Post

Correct. Which leads one to believe that according to them, amps do color and are not transparent. Given that, I ask how many people who believe in double-blind testing agree that all the amps they tested in 20 years, lack transparency.

Well, you can ask but I won't give the answer. I personally don't care much about how other people select their gear. I have my methods that appeal to me and leave other people to have theirs.

Quote:
Originally Posted by amirm View Post

Ah, you are making an assumption which we do not know to be true. What if that vocal person is dead wrong? You know, like the people who think "cables sound different." And he manages to convince everyone else that way. Now we have a "blind test with population >> 1" claiming results that are only represenative of the one person who is in fact mistaken.

Not sure I follow.

The test is performed as a series of multiple test iterations (to ensure statistically significant results) . During each test iteration the signal is first played through the "reference" and then either the object under test is switch in OR not switched in. The test panel now have to state wether or not they think the amp is in the loop or not. This procedure is repeated several times with the amp switched in at random iterations.

Since the test panel have no idea if the amp is actually in the loop or not during the second part of the iteration they have to rely to their hearing (and perhaps be subjected to bias from other panel members that state their oppinion). So, if the subject under test is in fact undetectable the answers will be random and uncorrelated to the actual test scheme. Any bias between members would show up as a correlation between test panel members, not to the test scheme.

Quote:
Originally Posted by amirm View Post

Bottom line is that we use multiple testers as to build up more confidence in the result, and lower the chance that a person voted mistakenly. By having people talk together, the testing validity reduces to much lower population. I would rank group bias almost as high as having the test non-blind.

Bottom line is that if multiple testers are used there's an increased chanse of someone beeing able to detect a coloration AND there's an increased possibility that one in the test panel just get lucky and guess right several times. Now, by selecting the number of test iterations carefully a result that is statistically significant even if just one person managed to pick out the amp can be achieved.

Now, deliberately picking out an amp that is truly inaudible as "colored" is not possible, since the subjects by definition will be unable to determine if it's in the loop or not so they will have to resort to guessing.

The potential problem occur if the amp is considered transparent by the test panel, since it is not possible to draw any conclusions from that outcome. It might be truly transparent or perhaps some "evil" person in the test group has managed to manipulate the rest of the group. That's partly why LTS for example is very strict with stating that "their test panel was unable to detect colorations during the test session", not that it is infact truly transparent.

H

hevi is offline  
post #208 of 914 Old 06-21-2009, 02:23 PM
AVS Club Gold
 
Art Sonneborn's Avatar
 
Join Date: Jun 2000
Location: Battle Creek,MI USA
Posts: 22,302
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 12 Post(s)
Liked: 44
The the statisitcal power can be increased with larger sample size (I guess panel members) why not the simple experimental design with a larger number of test subjects ?

Art

My HT


iRule rules my theater
 

"If she's amazing she won't be easy,if she's easy she won't be amazing"

 

Bob Marley

Art Sonneborn is offline  
post #209 of 914 Old 06-21-2009, 02:29 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
If the level of coloration is slight, then some people may hear it, others may not. You want the data to reflect that. If you let people talk with each other, then one party potentially influences the vote of the others, backing out the reason we use more than one tester. There is a big difference between "all four people heard a difference" and "one person heard it, the others couldn't be sure, or didn't hear it."

The caveat that "their test panel was unable to detect colorations during the test session" does not back out the distortion above. Instead, they should say, "the panel, despite the potential to try to convice each other of certain outcome, still could not hear a difference." I bet if they put it that way, far fewer people would accept their results .

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
post #210 of 914 Old 06-21-2009, 02:31 PM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,009
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 686 Post(s)
Liked: 388
Quote:
Originally Posted by Art Sonneborn View Post

The the statisitcal power can be increased with larger sample size (I guess panel members) why not the simple experimental design with a larger number of test subjects ?

Art

Expense and time. I recall that anything > 100 participant cost about $25,000 to $50,000 to conduct. Recuriting that many people, screening them, motivating them to come and take a test, etc. all cost money. And if you have single proctor, it would take weeks to complete. Using more than one venue and proctor, would increase the costs even more.

Amir
Founder, Madrona Digital
"Insist on Quality Engineering"

amirm is online now  
Reply Ultra Hi-End HT Gear ($20,000+)

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off