Why Blind Listening Tests Are Flawed? Per Robert Harley! - Page 29

I will make a correction to my above claim of application of the scientific method. I do not any longer apply any kind of logical reasoning, information, or the scientific method when dealing with "tbrunet."

Not at all. We have the full copy of the Bryston test report and their methodology. Here it is again: http://bryston.com/pdfs/07/Swedish14BSSTReview.pdf. And this is what they say to sum up their results up to that point:

“To sum it up, normally there are lots of views, ideas and opinions regarding the character of the tested amplifier after the open listening.

That was not the case this time.

We were sitting in open listening for well over one hour, and no one mentioned a single word about any differences they either imagined or heard. Actually, that’s the first time ever something like that has happened.”



Well, per above, we have a lot more than that, and it is all in English.

I have not heard of blind tests using test tones. Such a thing becomes very uncomfortable quickly. Here is what we see in the report above:

“We kept on listening without any changes. We switched between B and A, sometimes rapidly, sometimes with longer intervals. We tried having the music running while switching, and to play shorter loops and switch so
that identical sequence was heard on B and A.”

So we can confirm that they were indeed using music. As for artificial load, I encourage you to read their methodology. While one can argue that it may not be 100% identical to the load in your living room, it is a reasonable approximation (they used a speaker crossover). I will note though, that if you demand 100% identical situation to someone’s living room, then few A/B tests will pass by that standard. And subjectivists can use the same argument against you as we discussed before.

No. Reading the reports, you see that people criticized them for using to “easy” of a load initially so they made the load harder to drive to represent a speaker that is that way. But in no case is it doing things like having a 1 ohm load.

They played music and compared the output of the amp to input. They used real people who I am sure would go deaf if they were listening to test tones designed to make equipment fail.

So if I went and organized a double-blind test and found differences, I should be prepared to be told that some of them were poorly designed? Putting aside tube amps, where is the list of poorly designed amps I am supposed to avoid? How about CD players?

Can someone explain the engineering aspect of a poorly designed CD player and amp?

If asking you to comment on blind-testing of amps with detailed reporting, published for years in papers and magazines is a “strawman argument and logical fallacy,” let’s cut the conversation because I don’t know what else to put in front of you that would be acceptable.

Your position has been consistent. Question is, how do you explain it against facts to the contrary? Either you are wrong, or the facts are. Either way, we need more of an explanation than “what might be” going on with the test and instead, addressing the points precisely as I have done. We have a rare situation here with real data that didn’t come from either one of us. Data that a respected company like Bryston took seriously. I wish you would too . I know I did…

But then how can we trust your intellectual honesty?

The article doesn't contradict anything we have been saying. DBTs are still the best way to test for "audible" differences between these kinds of products. That Swedish article would have been more useful if they had used DBTs instead of A/B testing. That they used A/B testing instead of DBT is problematic and only weakens their conclusions. Arguing that we said differences can't exist, or that other tests can't work sometimes (though in this case the paper is still apocryphal as it is not peer-reviewed and it does not eliminate bias - as such it is not stringently scientific), is just another strawman on your part...

I don't recall anyone saying that differences can't exist. If anything, we probably all agree that when dealing with improper design (as was the case with the Bryston before hand) there can be audible differences. How much the audibility translates to real life situations and media is questionable; thus, DBTs become even more useful. There is no such thing as a perfect component, so the goal is to design pieces of equipment that have defects well below the thresholds of human hearing in real world conditions.

Without eliminating bias, Harley and his readership have no idea if he is fantasizing the elaborate differences he claims to hear or is actually hearing something pertinent.

the main issue is the absolute nature of his (harleys) statement.

If he had said "why i FEEL that blind listening tests are flawed for high end product evaluation" then we would have snickered and moved on.

in the absolute nature he ignores the fact that in the realm of identifying a difference, an ABX where a respondant can answer; same/different/no f'ing clue! is the only way to know. there is no other way.

for preference... ok, we need longer term listening, more selections, etc, etc. so arguing it would be impossible.

Let me see if I understand this Bryston link.

The original amp was obtained from an owner. All we know is that it was a large one.
They noticed some coloration which they subsequentyly attribute in part or in whole to a non-linear frequency response above 10 kHz as measured into a dummy load. We don't know if the dummy load is the same or similar to the unknown speakers used. Regardless, they noticed that the FR was down a few dB's above 10K.
They had discussions with Bryston around the time the actual review was done which was sometime in late 2006.

Drawing upon Stereophile's reviews of Bryston amps we have the following.

Bryston 2B-SST (review done in 2008) - FR unaffected by their (NHT based I believe) dummy load. FR down down 0.4dB at 20kHz into 2 ohms.

Bryston 14B-SST (review done in 2002) - FR essentially unaffected by the dummy load. FR down 1.4dB at 20kHz into 2 ohms.

Bryston 9B-THX (review done in 2000) - FR essentially unaffected by the dummy load. FR down maybe 0.1dB into

4ohms (my guesstimate).

Bryston 3B-ST (review done in 1996) - FR essentially unaffected by the dummy load. FR down maybe 0.1dB into

4ohms (my guesstimate).

Bryston 4B (review done in 1992) - No dummy load. FR down maybe 0.1dB into

4ohms (my guesstimate).
Of those listed above, only the 14B-SST had IMO a signficantly different FR rolloff which JA stated may be audible.

After the discussions, they were sent a 14B-SST that allegedly took their comments to heart with respect to the output filter and other matters. The reviewer talks about FR being within 0.05dB from 20-20kHz. No graphical information provided. That unit's specs, measurements, and circuitry ought to be different from the unit reviewed by Stereophile. At least that's my take on it.

The Bypass Test is level matched within 0.05 dB from 50 - 8kHz, uses test signals, and anywhere from 3-6 listeners. From what I read, I take it 'bypassed' means the signal goes straight to the speaker. Otherwise the signal is loaded with circuitry simulating a 'semi-difficult speaker' before being sent to the speaker again. Maybe it's my interpretation, but I'm confused by this. I'm reading it as the signal essentially either goes straight to the speaker or it goes through a circuit and then a speaker. This is peculiar to say the least. Am I misunderstanding something here, amirm?

The bypass test does not as far as I can see attempt to differentiate matters using musical sources. Rather test signals or probes. Hence, it is unknown to me at least whether the differences noticed translate into real world situations. I am also assuming that during the bypassing, the listeners know whether A or B is playing. Hence, not blind, right? The only thing that is blind is whether A or B is bypassed, yes?

It is interesting and I'd like to hear more about this. With all due respect in English though!

That really does not seem to be what he was implying. He was asking why would some seem to assume that more expensive is better. You refuse to consider that, and instead twist it around to something that you can provide a more self-serving answer to.

More expensive gear may be better, or it might not be. If more expensive gear were inherently better, that would mean that one could elevate the quality or performance of a component by simply raising its price. That's silly, Amir. Anyone who would assume universally that more expensive = better is gullible.

I ask because you implied that a 300 HP engine costs more than a 200 HP engine. If one changes the price relationship of the engines, does that also therefore change their power output?

Great post, Earl.

What is your interpretation of what that means? Since it says "either imagined" it doesn't look to me like it means that they found coloration with every amp except that one. They may have and they may say that elsewhere, but from what that says, I don't see that as meaning they found coloration with all the rest.

--Darin

my trake reading the swedish article was that he thought he heard something on listening, they did a blind AB and it was easy to detect using music and NOT test signals. then he took a closer look at the schematic and saw something that seemed like it might be problematic.

THEN he put it on a test bench and confimred what his listening told him.

he then contacted bryston expecting to be blown off... he was not. they saw it as well and updated the amp. in testing the new one, both with teat gear and both sighted and then blind listening, nobody could tell. those who chose to try (he said that only a few actually were brave enough to try to hazard a guess) there were 6 listening though. they could tell nothing. it was as transparent as they had ever seen.

All the test gear told them the same thing. i think that they did not include graphs on it as it was not crucual in understanding of the issues. and it seems like this was written up for bryston (in english) after the fact, so that may have been an editorial decision by bryston.

his point on the "test sounds" was that after nobody could even flip a coin, they decided to bring out the test noise (that is the 800 pound gorrila to amps) that is almost never used. and even with that, they could not detect a change.

all in all i feel reading that article that their process, integrity and results are top notch.

i am very fond of listen first, and if you suspect you hear anything, then use the test gear to try to help you identify it. i also really like that they did the sighted comparison first so that the group could work together to identify things. (the unknown thing i thought i heard might be better identified in consultation with others) AND THEN... they did the blind test.

so in reality, it combines both subjective and objective results together for a result that cant really be knocked down. notice that all he evern claimed was comparison.

for preference, it took long term listening and comparing to others.

That's fine Dizz. Can you explain to me the chain of components in bypass vs. non-bypass?

"Scotty, I need more power to the defense shields!"

Heh, heh.....he said "teat"...heh heh, heh heh

Am in a meeting but here is the test harness: http://www.sonicdesign.se/amptest.htm

Thanks, but I'm still unclear how the Bryston amp was tested. I'll take any explanation I can get.

So, they are testing an amp by listening it through another amp?

Frankly, that's the most absurd amp test I can think of.

It would be like wanting to show that AVC is transparent to the source by comparing the output of an AVC encoder fed by either another AVC stream, or by the uncompressed source.

I don't think that *you* would give much credit to such a test, would you?

Correct.


Wonder what NIN has to say about this .




Seriously, there are two ways to look at this:

1. The test harness amp (not the one under being evaluated) has so much coloration itself that it totally masks any difference between the input and the output of the test amp. If this is the case, then I think we have already proven that amps do have performance differences. And further, they would not be able to hear any differnces yet they clearly are.

2. The test harness amp is transparent with respect to showing the difference between the input and output of the amp. In which case, their results are valid in how they show amps to color.

So as you see, all roads lead to Rome!

If it were me, I would have used my headphone system which I know to be more transparent than any speaker setup they have.

3. They both have inaudible issues that compound and become audible...

So I take a speaker wire which has inaudible issues, and add that to amps which have inaudible issues and I have audible distortions that show up in blind tests? If so, subjectivists have proven their case again that wires do make a difference as do amps!!!

Say it ain't so. NIN, get your ass over here and explain this thing!

I erased my posts. I'll let someone else argue with Amirm; it isn't worth my time...

Adios (*unsubscribed*)

I don't see why this is at all unreasonable. The addition of subthreshold differences can combine and produce greater differences that are audible. This is certainly true of visual perception of colors, and aggregate audio changes could certainly grow to become audible.

I'm one of those that feels it's worth getting decent gear in the hopes that it all doesn't add up to something annoying or unsatisfying. Like decent, but not expensive, interconnects.

It's difficult for me to argue against you, as I can hardly claim your expertise. But I do know that I try to be honest in my assessments of equipment, and, ever since moving from Citation amps to Krell amps, and then on to Halcro amps, I've maintained that amplifiers are as important as the preamp or processor. Certainly, I can tell a "signature" sound in an amplifier, that, as you say, I may simply "prefer."

I had the opportunity, however, to listen to a two-channel setup of Halcro high-end amps (with, I think, a Halcro preamp), through some Magnepan speakers that I've heard many times before. The room lit up with sound in a way I'd never experienced before. I was not just hearing things. The sound was palpable, and I'd love to be able to afford those $70,000 amps.

I defer to your experience, but it has not been my experience. Amps matter, to my ears, at least.

Thanks,

Nick

So the speakers which you had heard many times before it sounded much better in this setup than any you had heard previously. Nick, is why did you decide to attribute it to the amplifiers? Why for instance, didn't you instead conclude "this is the best sounding room I have ever heard these speakers in". You are not sure of the preamp model - maybe it was the preamp (I don't think it was but the point is you could just as easily have attributed it to the preamp). The room that you were in, and the placement of the speakers in that room, likely had a 100x greater impact on what you heard than whether a Krell or Bryston or Halcro amp was used.

Such anectodal experiences have as much meaning as the water tasting in this.

Let me be provocative but quite serious for a moment:

How many people like to close their eyes, dim or turn off the lights, or do something similar when listening to music. Do things sound better this way? Is the imaging improved?

Most importantly, if so: is this a real or non-real effect?

This is a rather serious question, and I'm curious to hear what people do and why. And particularly whether this is "real" or not.