How do I verify - or debunk - the claims of The Upgrade Company? - Page 36

If he agrees with that, then I have no problem with his stance.

For me, this has boiled down to a very simple premise. Blind ABX testing is the gold standard approach, and if it shows a difference between two products, then we can be confident that the difference is real. In other words, its good at proving that there IS a difference.

However, its a tough test for audio comparisons, and I'm always hearing about some equipment / modification / reviewer that can't pass a blind ABX test. Maybe there's a highly-trained pair of golden ears somewhere in Harman International who can hear subtle differences, but the ABX test is otherwise extremely difficult to pass for audio electronics. There are lots of well-known instances of this:

• The Richard Clark Amplifier Challenge is well-known. He will pay anyone $10k if they can hear the difference between two amplifiers - as long as they're not driven into clipping. That's ANY amplifer. He's pretty confident about that, and I believe no-one has succeeded.
• http://www.tom-morrow-land.com/tests...bcGTlHNM5H4jfQ
  
• BRAD MEYER and DAVID R. MORAN published the results of extensive research called "Audibility of a CD-Standard A/D/A Loop Inserted into High-Resolution Audio Playback". They concluded that no-one could hear the effect of 16/44.1 conversion, unless the volume was so high that the noise was audible. (Even though all the sound engineers on gearslutz, to a man, appear to prefer the sound quality of analogue recording to digital. Interesting site, that; thanks for the link.)
• http://drewdaniels.com/audible.pdf
  
• Peter Aczel, "The Audio Critic", is a fierce oponent of subjective audio assessment, and insists that audio equipment must be assessed objectively. Measurements are best, but if you do it subjectively, then it has to be scientifically. Subjectivists hate him, but his position is that all properly-designed amplifiers sound the same.
• http://www.theaudiocritic.com/&sa=U&...HlSXV8SLrn0CWg

What do all these examples have in common? They all require that listening comparisons are done with blind ABX testing. I hadn't looked at this very closely before, but I get the impression that ABX testing is very good at proving that there isn't a difference, but its not good at proving that there is a difference.

As far as I can tell, ABX always shows that audio electronics sound the same. So if that's what you've decided beforehand that you want to prove, then ABX testing is the way to do it. And nobody will ever be able to contradict you.

Hence the question in my first post here - has anyone heard of any instance where an ABX test gave a positive result with an amplifier?

Nick

Not completely accurate. ABX testing is merely a means to gain perspective. You cannot prove that there is no difference--that would be an absolute. What you can show is that what one believes to be true subjectively is or is not accurate. If you review two pieces of equipment subjectively and identify differences, then do the same in a DBT, and find no differences, which is correct? Where there differences that you couldn't see/hear the second time? Or were the differences imagined?

But a single or even multiple tests will not make the conclusion absolute.

It's been used extensively in evaluating codecs for 'transparency'. Search in Hydrogenaudio.com as one place.


Several years ago, Rega released a CD player, The Planet. They touted the use of a proprietary DAC (later determined to be untrue but other companies used the same DAC). Subjective reviews were generally quite positive as they're wont to be in the audio press. However, some people quite lambasted the product resulting in heated words as you might expect. Some publication in England I believe turned to blind testing to see what was what. Turned out that the quality of the sound depended upon the preamplifier and performance issues were heard with a particular Creek one.

So, they dug a little deeper. Turned out the DAC (no longer made or recommended by the manufacturer), had issues with with distortion products past 20K. One might say so what since we can't hear up that high and besides there's precious little musical information past say 17K in modern recordings. Well, the Creek preamp, which performed well 20-20K, didn't do so well once you got past 20K and when fed signals that had +20k content resulted in IM (intermodulation products) that were located in the audible band. Hence, one would hear grunge, but change the preamp to a well behaved one and no problems.


I'll give a very simplistic example, but Richard Clark would have a way to compensate. Take two amps. Let one be solid state with a comparatively low output impedance. Let the other be a SET or an OTL. Those generally have quite significant output impedances. Now take a speaker that has significant impedance variations. Level match at say 1KHz. You'll hear differences.

Did you mean David Clark?

Not if they actually sound different. For example, tests have shown that an amp will sound different from the same model amp with a few tenths of a dB difference in broadband level.


Incorrect.


Yes.

There are many examples of blind testing including rapid switching that generally yield valid information. For example, if anyone has had their eyes tested they know the whole better/worse thing. Would you really expect to get the right/best prescription with long term looking? Look up how the planet (I still call it a planet) Pluto was discovered. Food companies often use a variant called triangle testing when evaluating alternate sources of raw materials since they are concerned about customers complaining that the taste has changed. In this type, three samples are prepared. Only one is different and the objective is to identify the different one. Trained and untrained people are used.

Companies that make fabric softeners, well some of them, use blind testing to determine degree of softness. And so it goes.

The "audio is different" or unique claim is interesting. In a debate I had with a subjectivist, I compared audio claims with those made for Penta water, a product that's supposed to be "superior" to "regular" water, based on pseudoscience. The claims are accompanied by glowing testimonials from "satisfied" customers. Sound familiar? The subjectivist's response was to sputter his indignation that I would dare compare audio claims with that sort of scam, or any other claim that smacks of the "paranormal".

Video is "different" as well. Richer blacks and sharper colors attained from a digital signal.

RobertR -

Speaking of the paranormal, psychics have been known to dismiss the validity of blind testing with respect to their own claimed abilities. After having failed such rigorous scrutiny in the past, their defense is that traditional science hasn't yet developed a means to reveal their genuine psychic powers; that these unique abilities get lost in the process of being subjected to the DBT method, so there must be a flaw inherent to the method itself. Sound familiar?

all video parameters easily measured with the correct equipment... so no, it's not "different" at all...

i'm not even gonna touch the second sentence...

I think txmxer is only restating the claim offered up by at least one customer of TUC within this thread. Or maybe the claim is being made on the TUC website. Whichever the case, I'm pretty sure it's not the view held by txmxer.

^^^

ah gotcha... i have problems sometimes remembering who plays for which team...

Absolutely! I was amused at the indignation of the audio subjectivist for daring to suggest that an audio scam is really no different from any other claim based on a lack of sound science or a denial of the validity of science itself.

Houdini tried to get them out but you know, they have rights.

Let's go back to this comment for a second. Back in the good old days (early 1990s) the higher end audio stores would allow you to take home speakers and try them out. If you weren't satisified you could upgrade to the next speaker in the line. It was a great way to get the customer to keep buying more and more.

Of course the basis for this was that a speaker interacts with the listening room. Every speaker has a certain amount of interaction with the cubic volume of air and the reflections in the room. No amount of work at the store could reproduce how a particular speaker would sound in *your* listening room. This was an acknowledgement of that difficulty and most of the time people would upgrade because the grass is always greener (or sounds better) with the more expensive speaker.

Now with Blind A/B testing the intent is entirely different. The idea is to simply follow the scientific method (!) and only change one variable at a time. The only difference is a change in one component. Even the time variable is minimized.

So equating AB testing with how the overall system interacts are two separate items. How the overall system interacts with the listening environment is important (and likely improved with speaker positioning and room treatments). However that has nothing to do with the question of whether component A really has a different output than component B, which can be measured (for an absolute comparison) or Blind A/B'ed (for a relative comparison). The intent of the Blind AB test is to narrow down the differences to one component (change only one variable at a time).

The difficulty of using the "must listen in my room" argument is that your room is constantly changing, too. Air pressure differences as fronts move through will change the way sound is heard. The listener's mood that day will change a subjective opinion, too. Was the listening done in the morning or evening will change perception, too (dead ear syndrome)? How is the ambient noise? If all we could do to judge audio components was listen in our room, then we would have no way to narrow down the range of components except to purchase every component out there and try it. I don't think my wife would allow that...

I was just going to say the same thing... Jeff and I have seen in our *limited* A/B/X testing between an Onkyo 885 and 5508, that we CAN reliably ID between those two units.

My opinion is that the difference between these two units is audible in an ABX test. The A/B/X difference between the 885 and 5508 is audibly MUCH greater than any differences we thought we heard between the stock 5508 and modded 5508 - even before the blind test, IMHO.

That's certainly not true. There are differences in equipment, as anyone would expect. Even true for cables and interconnects: nobody is saying they are all the same. But what is undeniable is that if they all measure the same, then they ARE all the same and will sound all the same too.


I can't recall one but it would be very easy to set up a test that ABX-ed a tube amp and a SS amp. The differences between the two are easily measurable and would therefore show up on a test. Note that this does not apply to ALL tube amps and ALL SS amps. It is quite easy to design a SS amp that mimics the sound of a tube amp, for example. And, of course, once one has determined that amp A sounds different to amp B and can reliably be identified as X with far more certainty than chance alone can determine, then one can decide which one is *preferred*. Some may prefer the tube amp, some the SS amp - but the first step has to be to be able to reliably hear a difference.

The problem that the Upgrade Evangelists have is that they deny there is a test in the world can show a difference, and this is how they explain that they cannot reliably hear a difference! They have already decided there MUST be a difference (because someone says there is and charged them a lot of money for it) so if the test doesn't show the difference, then the test must be flawed. It is never that there is not a difference, always the former.

It's why this switch nonsense keeps coming up. It's something they can latch on to to 'prove' the test is flawed.

What do they think the switch is actually doing? They don't know/say.

If you could show them a simple test where one measured the input and output signal of the switch and showed them to be identical, would they still object to the switch? Of course they would!

If you say you will eliminate the switch and swap cables manually, then they have an objection for that too. So the test can never win. It is always wrong if it reveals no differences between the stock unit and the modified unit.

You can't prove a negative, because that would imply that you've tested every case. All it would take is one solitary case where someone could reliably hear the difference to establish that there is an audible difference. If you can't find anyone that hears a difference, that doesn't mean you've proven it doesn't exist, just that you haven't found anyone that can hear the difference unsighted (yet).

That's right. It's incorrect to say "the claim is presumed true unless the ABX test proves it isn't". The burden of proof is on whoever makes a claim. If someone claims "component X sounds different", a failure to hear such a difference in an ABX test means that the claim is unsupported.

What's the posited cause of the difference?

Is it a different version of Audyssey which leads to differences in FR?

A technical flaw in one of the units (older or newer) that does not appear in the other?

Frankly, nothing except for those two reasons would make sense, based on our accumulated knowledge of sonic differences in audio electronics.

Shouldn't we draw a distinction here between amplifiers, which really should all do the same thing in an indistinguishable manner, and processors/receivers, which do apply algorithms and such to decoding sound?

I totally buy the argument that solid state amplifiers, unclipped and running within their design parameters, should all sound the same. I'm not at all sure I accept that of the processor circuitry.