How do I verify - or debunk - the claims of The Upgrade Company? - Page 46

Very interesting thoughts Craig... I know ABX is commonly regarded as the gold standard in audio tests, but you may have a point that the ABX test may not be a "high sensitivity" test - which may account for results that say two DUTs sound the same, when there really is a measurable difference between them.

Just how did this switch become controversial?

It depends on what that difference is. Not all measurable differences are audible or, even, would be expected to be audible.


It could be that the difference is, in fact, inaudible or that the test conditions obscured an audible difference.


In principle. How it is set up and executed is crucial.

IMO before you can ask which you prefer you have to prove a detectable difference. There were, especially at the beginning of the digital age, amps tha measured differently in a way that was recognized to be likely audible. A fair number rolled off the highs a bit starting around 10 KHz. Measurable and audible. But comparing an amp that rolls off the highs around 25 KHz to one that measurably stays flat out to 50 KHz becomes a different matter. One needs to prove the "flat to light" amp is audibly different from the one with a rolloff to prevent oscillation.

Typical double blind testing starts with a null hypothesis like "these 2 devices sound the same" and can either disprove the null hypothesis or not. They cannot prove the null hypthesis. Although theoretically enough comparably administered tests achieving the same result would start to get you there statistically.

Questions about preference really are relevant only after a difference is established. And as Coca Cola proved to itself, there may be unanticipated nuances to the question of preference. Like folks who prefer a sip of Pepsi may prefer the less sweet Coke taste if they're taking more than a sip - like a whole can or something.

I placed quotes around "controversial" because there are two diametrically opposed viewpoints on this. There is a group of people who think the switch is the culprit device in this blind test. This group has been collectively defined as the "believers". There is another group of people who we can collectively define as the "non-believers", who don't think the switch is controversial at all. They feel the switch is obviously and completely "benign."

Nonetheless, the simple fact that there are two groups who disagree about the use of the switch makes the the use of the switch "controversial."

Simple as that.

Craig

So Kal, does this "verify - or debunk - the ABX test...?

...or even THIS ABX test?

Craig

There is nothing more sad than seeing the religious on a crusade.

The switch allows for more rapid comparisons enhancing the ability for an individual to ascertain small differences. Manual switching of cables introduces significant time delays thereby reducing discriminatory abilities. One could easily conduct a scenario to test/prove the ability of a switch to allow a user to achieve smaller detection limits. If I wanted to cripple people's abilities, I'd eliminate the switch and introduce increasing time delays.

Just to be clear, I'm not saying that it's not a high sensitivity test. I'm saying I don't know if it is a high sensitivity test. If there is conclusive evidence that it is a high sensitivity test, I would like to review that evidence.

Craig

I do not think these specific issues inform much because we do not know if there are any real and measurable differences between the two DUTs.

My primary criticism of the current ABX test is that its sample size is too small to inspire confidence either way. It "suggests" that there are no audible differences under the described conditions but I cannot stand up and defend it.

If you are referring to me, I am absolutely NOT the religious. I don't personally "believe" these TUC mods make an audible difference. Nonetheless, putting my own beliefs aside, I am only asking that science be appropriately and honestly applied to this discussion. I don't think we can give "the switch" a free pass just because we, the believers, *think* it's benign. I don't think we can give the ABX test a formal stamp of approval just because it agrees with our pre-conceived notions. I don't think we can accept the results of 3 listeners, listening to 11 samples of content as a "broad generalization" that there are no audible differences. I am only asking for a more rigorous and thorough evaluation.

Craig

Quite the pity that DS and TUC are not forthcoming when it comes to the results of their DBT's that they claim to have done.

That's the wrong test IMO. If I'm going to spend thousands of dollars, I want a test with a high specificity - IOW, a low false positive. A test with a high sensitivity but low specificity would make it highly unlikely that there was a difference between units and it was not detected, but would have a high likelihood that where a difference was detected - one doesn't actually exist.

Although it is true a test with 100% specificity and sensitivity likely won't be found, tests with high sensitivity OR specificity do frequently exist. Choosing the proper test for the application is what's important.

Its why in medicine tests with high sensitivity are used for screening and highly specific tests are used for diagnosis/treatment. For example, a test with a sensitivity of 99% and specificity of 30% makes a great screening test since its unlikely someone with the disease will test negative. But many without the disease will test positive. That's why you wouldn't want to get treatment based on that test. Before treatment you want to follow up with a test with a high specificity - or a low chance of a false positive- to be sure you actually have the disease.

The "treatment" here is the money leaving my pocket. And for that I want a test with a high specificity - or a low chance I heard a difference between the two units when no difference existed.

The problem is that sensitivity/specificity are scientifically determined and that data simply doesn't exist in audio as far as I am aware.

For the record, the Pepsi Challenge was tainted anyway. I watched the testers pull the Pepsi's from a cold ice chest while the Coke's were at ambient. This was only one location, but I wouldn't be surprised if it was common practice.



This thread has become interesting again. ABX testing does have it's challenges. In and of itself, it creates a bias if one knows they are being tested for something specific. Anyone that reads all of this would be a less than ideal candidate to participate, feeling that their sensitivity to audible differences was being tested.

Ideally the people that participate in the test wouldn't have a personal connection to the issue.

In your experience, you expected one thing, but in the test, you had conscious or subconscious pressure to perform--to be able to repeatedly hear the differences and choose the right product. To hear the differences. This could raise your BP and alter your hearing in the moment.

I am a fan of scientific methods, but not a fanboy. We have to be honest with ourselves and the limitations of testing methodology in order to be true to it.

Kal, I have to ask again (in getting back to some of the earlier claims), if the difference is so small as to be obscurable by the test, doesn't that make the value of the upgrade, at best, so small that other upgrades are a better solution? In and of itself, isn't an "obscured" conclusion a negative result since there are other upgrades which provide tangable improvements?

I guess if the audio world was out of other upgrade possibilities then an obscured result would be significant, but that isn't the case (luckily).

It really comes back to a value measurement - is such a small improvement (if any) really worth the apparently large amount of money to be paid by the future owner? Or, am I missing something?

Out of curiousity what other upgrades (mods?) are a better solution that provide tangable improvements? When you say tangable improvements would these upgrades (mods?) be proven such as through ABX testing?

Bill

In that case, all ABX blind tests conducted to date are flawed. That seems most unlikely to me.

I think the biggest flaw in the test conducted wrt to this specific thread is that no objective measurements were taken of the two units under test. If measuring equipment had been available it would have been a straightforward matter to test unit A and unit B and to see first of all if there is any objective difference between input and output of both units. If there is no difference, then the two units will perform identically. If there is a difference, then the question becomes 'is the difference audible?' and finally the question is 'which of the two different presentations do I prefer?'.

The listening tests can then be compared with the measurements and a check made to see how the measurements corroborate, or not, the audible differences. For example, if the measurements show an extended HF response in one unit, and that one unit is always observed as being "more open and transparent, more airy, in the treble" then one can see that the listening test results are grounded in scientific fact. If OTOH, nobody can hear any difference which would corroborate the extended HF in one unit, then one can come to the conclusion that the modifications did indeed make a difference (extended HF) but the difference had no audible effect.

I am not knocking the test conducted and I know how difficult it is to cover all the bases, especially when someone suggests using very expensive test gear which may not be available to the testers.

The problem is always going to be that the aficionados of the modifications will always point to flaws in the test if the result does not favour their modified unit. If, of course, the results did favour their modified unit, then the test would be 'perfect' and would 'prove the point'. Such is the nature of subjectivity and subjectivists.

My own personal take on this requires no further tests at all: the customer is promised a hugely significant upgrade in performance, such that his modified unit will equal or better competitive units costing $10,000 or even $20,000. That there is so much discussion as to whether any difference in SQ at all exists, indicates to me that the promise of the manufacturer has not been met. If he had achieved his goal and fulfilled the promises on his website, the audible differences would be heard by all. If they do exist they are simply too small to warrant the price asked and the equivalent money would bring greater returns if spent elsewhere, on proven modifications that can be easily demonstrated to bring significant improvements to SQ - eg bass traps.

Exactly. The switch is an issue solely because it was a straw to be grasped when the tests did not produce the desired result for the 'believers'. If the cable swap method had been used, the objection would have been the same, for different reasons. A no-win situation for objective testing.

I agree. The switch needs to be *proven* to be benign. This is not difficult and I would think that the manufacturers of these switches would provide the proof that their switch is not adding anything or taking anything away when it is used. Once that has been established, then the switch becomes a non-issue.

Of course, it is highly likely that the believers will question the switch manufacturer's own test methods, results etc. The switch, for them, is a straw too useful not to be clutched at.

Bass traps and room treatments come to mind. Easily measured differences, easily heard improvements. Audyssey XT32 is another. The same amount of money spent on either of these, as opposed to the TUC modification, would bring tangible results that can be heard AND measured.

I also agree with the poster above who says that if the differences are so small as to generate all this controversy, then one has to question whether they are worthwhile, even if they exist. There are no threads endlessly discussing whether room treatments offer real or illusory benefits, for example.

Some of the TUC supporters might like to hold the switch accountable for the blind test result, but I'd like to go on the record to distance myself from that idea, which I think is clutching at straws. Yes, every cable, connector, switch or whatever will cause some degradation to the sound, but in this case it will be tiny, dwarfed by everything else, and almost certainly inaudible. If the switch made a difference to the blind test, then it would have made a difference to the sighted comparisons as well, wouldn't it?

The big issue for me is the ABX test. It seems that the ABX test, although scientifically valid and desirable, can be used to indicate a null result with almost any two pieces of audio equipment you chose. That doesn't mean the test is wrong, and I'm not taking anything away from what Meyer & Moran or Dennis & Jeff have done, its just that it isn't as sensitive as other, less robust, tests.

The significance of the ABX test is that where it gives a positive result, you can be very sure that its a genuine positive - as long as the test is done right, false positives are very rare. I don't think you can criticise the ABX test for giving false negatives, though, all the tests that have been performed haven't all suddenly become flawed.

What it doesn't do is identify smaller differences - in particular, audible differences, which are necessarily sequential comparisons, and depend on memory. I guess that our audio memory just isn't that good, and if we want to identify smaller differences, then we need another way. The problem is increasing the sensitivity of the test without having too many false positives. Maybe it would be possible to equate the profile of success % vs sample size for ABX, with success vs sample for some other test, like an AB test?

So if you need say, 8/10 in an ABX to get an arbitrary confidence level, could 9/10 in an AB test give the same confidence level? The numbers are illustrative, but you get my point?

I realise some people will insist on an ABX test, and they can continue to prove that everything sounds the same, if they chose, but I'd be open to some relaxation of scientific protocol. Its not like my life depends on it.

Nick

Keith,

Excellent post. I agree and the above portion of you post is totally being dismissed or ignored for the most part by the TUC supporters. TUC's grand claims are not being fulfilled by the results of the ABX test conducted by Jeff and others. Even some of the posts by those that own TUC modded gear indicate the SQ improvements are subtle. So why are the TUC supporters not questioning DS's grand claims of huge SQ improvements when they are actually subtle or possibly non existent? I know if I paid $1399 to have my 886 modded I would expect the huge improvements that DS claims. If not I'd be asking for my money back.

Bill

I've made the same point several times, Bill, and it is never taken up - presumably because there is no answer to it that they can provide which makes any sense.

The always-reasonable Nick, in the post almost immediately above this one, for example, is discussing "small differences". But the manufacturer is not promising SMALL differences - he is promising absolutely HUGE differences - such that the performance of your modded 886 will come to sound as good - or even better - than units on sale elsewhere for up to $20,000!

That alone, to me, makes the whole topic of the efficacy of the modifications null and void. The manufacturer is clearly not fulfilling the promises made and charged for. Isn’t that all that matters?

If you accept the test conditions as valid or, at least, reasonable, yes.

Hi Nick,

I'll have to disagree with you on this. At this point, nobody knows if the switch made a huge difference, a small difference, or none at all, so folks can only speculate. Tests would need to be done, in order to come to a more informed conclusion. For me, I wouldn't want the switch involved, because I'm not going to be listening with a switch in my home anyways. Cable swapping would be more than fine for me, and the short duration for doing the swap would not bother me either.

Dave