How do I verify - or debunk - the claims of The Upgrade Company? - Page 45

Plenty info here ,the recent post strike me as some personal discussion because the facts as reported are duly noted and been thoroughly beat to a pulp.Please post new info if you really have something important to say, the same thing typed 10000 times gets old.

Interesting link with an expected result.

What we learned from the test of the TUC modified unit was that the three people involved in the test could not identify the modded unit under blind conditions. Would the same conclusion be reached if more tests were conducted? I have my suspicions but who knows.

But I also agree with another poster who suggested that if a series of blind tests were conducted with the most stringent set of parameters and a much larger sample were utilized, and that test also yielded a result of not being able to identify the modded unit, those who want to believe would continue to believe and somehow blame the testing procedure.

Just go read other forums where these kinds of discussions take place. The believers will continue to be believers no matter what evidence is presented to the contrary.

I agree. I have my suspicions and expectations too.


I also agree with this. We're not likely to change believers' beliefs by presenting them with scientific data. We're even less likely to convince anyone of anything if we try to portray a limited set of data as something more than it is.

Craig

Yes, you guys are right. (Sometimes, Bill, I need telling twice ). I have made certain adjustments that will help in this regard

The tests they did on their ABX switch (linked to on the review page) show conclusively that the switch cannot be influencing the result, which is what I was calling for earlier.

Their conclusion after testing the switch is: "En esta gráfica podemos observar el perfecto comportamiento frente a una señal digital de formato AES3 con muestreo de 48khz. Por tanto, podrÃ*amos utilizar el ABX para conmutar señales digitales de audio hasta un máximo de 56khz de muestreo si diera el caso."

The "perfect performance" (el perfecto comportamiento) is the telling phrase.

HST, I liked their "human ABX" too - maybe that would be a way to go in any re-test? Would the defenders of the mods find that acceptable I wonder?

"Science"? Not much "science" exhibited in this thread. No lack of "speculative extrapolation" however. Might be time to change the name of the thread to the "speculative extrapolation" of conjured up "science" relating to TUC.

move on please: if you don't like the thread, please hit unsubscribe

otherwise please limit further posts to technical issues

In a debate I had with a subjectivist, he made it clear that the rest of the world could fail to hear differences, and he wouldn't care, because reality revolved around his ears.

I think each individual should feel that way. If the equipment/music make them happy, then that's all that matters--for them.

The conflicts only arise when someone chooses to profess the greatness of something without evidence.

non productive posts removed

think before you post here: will my post contain relevant technical info?

more: problematic posters will be asked to leave (thread ban) without further warning

Jeff,

We all have different standards for picking out different things from cars to food to audio gear. My standard for picking out audio gear is not a blind test. Some folks like measurements, some a form of blind testing, others listen subjectively, and some like combinations and variations of those.

I enjoy reading all of the viewpoints and the data from any/all of them. What I do not enjoy is someone forcing (perhaps another word would do better here) their standard upon me. If blind testing is what you and other require - more power to you. I would never want a switch involved for testing, but if others do, then that's their choice. If one blind test involving a couple of folks is what convinces others about the validity of something, then that's their choice. If others decide to try it anyways and like it, then that is their choice too.

I think you have really great and honest intentions Jeff - folks should really be appreciative. I think it's great that you've shared your experience and your resulting opinion from the test that you did. You've put a lot of time and effort into the testing and into this thread! However, it's just not the answer for everyone - I know it's not the answer for me.

Yes, please share what you've found and how you feel about it, but no, please don't feel that you need (or that it's your responsibility) to make it everyone's choice. It's not the Design of this forum to do so.


Dave

I think this thread is the most worthwhile of the TUC threads, as it includes independant listening, and blind listening at that, which moves things on a lot from what's been discussed before.

However, before it goes the way every other TUC thread, I'd like to return to the key element of the blind comparison - not the fact that its blind, but that its an ABX test. Although I'm obviously a TUC supporter, and one who'd be reasonably confident to identify a TUC product in a blind AB test (like Rich), I'm pretty sure I wouldn't in an ABX test. So do I have to reconsider my view on TUC?

Although its the gold standard for such testing, ABX seems to be the graveyard of all audio comparisons. There are some people who have succeeded in ABX tests, but they're the exception to the rule. The example of the Molingordo IV tests shows what happens even when A and B are very different indeed - almost representing the extremes of the audio world. Very difficult to believe that they really sound the same.

I've been reading the Meyers & Moran paper again, like David Moran says in the early debates - read the paper! And I wish I'd been more thorough last time. I presumed that what they'd done was take HD audio and truncated it to CD audio, and then do the conversion and amplification. But no, they instead took the analogue output from a CD player, converted it to 16/44.1 digital audio, then converted it back again. The direct feed from the SACD player didn't go through any A/D/A conversion.

That pretty much seals it for me. I'm ALMOST prepared to accept that 16/44.1 audio is transparent as long as the mastering and conversion is well done (though its a hard sell), but the idea that two D-A conversion stages are transparent is too much to accept, nomatter what the science says.

I've been listening to transports, players, DACs, processors and amps for decades, and the one thing that really bottle-necks a system is the component that does the D to A conversion (I'm trying to distinguish the DAC equipment from the DAC chip here). And if you add a double A-D & D-A converison into the equation, it sticks out like a sore thumb. I've used several Arcam, Lexicon, Anthem, Proceed and Meridian processors, and even the best of them sound mid-fi when you use an analogue input that is digitised. And those are usually 24/96 ADCs and 24/192 DACs.

So the idea that an A/D/A conversion process is undetectable is too much for me - never mind the fact that its a 16/44.1 process. Sure, ABX DBTs have been used to show that interconnects and power cables all sound the same - I have no problem with that. For a test to be useful, it needs to have a level of discrimination that's useful for what you're trying to assess. No point having a test that's so difficult that no-one passes it. That's like giving a degree-level exam to an 11 year-old; what does that prove?

So just imagine for a moment - WHAT IF YOU ABSOLUTELY, POSITIVELY KNEW that there was a difference between two amps - but the test said they sounded the same?

That's why I'm so keen to hear how Jeff & Dennis get on with the 886 / 5508 ABX test. My money says they'll sound the same in an ABX test, but as I said before, I'll eat my words if proved wrong.

Nick

^^^

yea, i'm a bit perplexed by that...

either it exists, or it doesn't...

Hmmmm, well, I did say imagine for a moment, like doing a thought experiment.

I'm not sure. What I would probably do is perform a number of blind AB tests instead, and try to use those results to show that there was a reliable, repeatable and identifiable difference. I haven't looked at the rationale for using ABX tests instead of AB tests, but I'm sure there are good reasons. I'd just like to find ways of using blind AB tests to give an acceptable level of confidence in the result. I suspect that AB tests would be more discriminating.

The other approach might be to knobble one of the amplifiers (or whatever) by introducing a subtle fautl, like resetting the bias in the output stage, that would still give an apparently similar sound, but would nevertheless be identifiable from measurements (and hopefully by listening). Another might be to add a 12-bit A-D-A conversion stage in the the line.

Nick

Nick, is it that your thought that multiple A/D/A...and so forth conversions ought to be audible when comparing say the first analog signal to say the second or third?

http://www.avsforum.com/avs-vb/showt...5#post21863335

If you could spot the difference instantly but not with a break does the difference not matter? My thinking is until the test catches the differences it's not valid?

I haven't really thought this through yet, but I guess what I'm after is a way to establish a level of sound quality degradation that is audible, but which can be verified without using an ABX test. And then perform the ABX test.

Performing repeated A-D-A conversions might be one way of introducing an audible change, and a tone control adjustment should be easy, shouldn't it? Another method might be to simply add distortion - include a variable compressor in a circuit, and adjust it up and down in real time until you find the point at which the distortion becomes audible.

Suppose that level is equivalent to 1% THD / +1dB / 1 conversions or whatever; take that signal and compare it with a clean signal in an ABX test. If the difference isn't audible, increase the change to 2% / +2dB / 2 conversions, and so on until you get a positive result.

It doesn't sound easy, but it would be an interesting result. Just how much change or degradation do you need before its audible under ABX test conditions? None? Two times? Ten times? I've no idea; no feel for what the answer is, but I'd sure like to know.

I think the key is to have an independant but credible way to determine audibility thresholds.

Nick

Before we vacate this question, let's reconsider it for a minute...

What if a component measures differently, so that you *knew* it was different; yet in an ABX test, you couldn't reliably distinguish between the components? What would that say about the ABX test?

Jeff, earlier in the thread you speculated that DS was "goosing" the FR. You said you could hear it in the pink noise. If he was goosing the FR, that should be measurable. Yet, when it came to the blind ABX tests, you couldn't identify it. If those FR differences were real, what does *that* say about the ABX test?

Again, I'm not saying that the ABX test is bad. I'm just saying that, when using the scientific method, a healthy dose of skepticism is always a good thing. We *think* we know the switch is a non-issue... but is it really? We *think* we know the ABX test is the gold standard... but do we really?


That should be interesting as well. I hope to participate this time. It will be a blinded ABX test, so it should reveal the significant differences we have claimed to hear between the 885 and the 5508. If it doesn't reveal those differences, does that tell us more about the ABX test than it does about the differences in the DUT's?

Craig

It seems that you want to practice on something, so there's a few things you could try, Nick. One, would be to simulate something like a SET or OTL tube amp using an existing solid state amp. Those type of amps generally have high output impedances which you can sort of mimic by adding something like an 8 ohm power resistor to your speaker wires. Then, just level match at 1 kHz and see if you can identify which is which. Once you get good at that, try smaller values of resistance.

You could also download a free program called audiodiffmaker. Take those multiple A/D/A conversions and use the program to both time sync and compare something like the first analog signal to one down the road. It'll allow you to play the difference between the two.

I don't know exactly what the point would be of creating a mangled signal to use during a test, but putting a resistor on the outupt of a typical solid state amp can make enough of a FR difference to be noticeable. While multiple AD-DA conversions almost certainly would yield slightly different looking output, it may or may not be audible. I guess if you're trying to assess audiblility of AD/DA it could be a valid thing to do.

But using a compressor to "distort" the sound is not going to tell anybody anything except that you can hear compression once it reaches a certain point. You have an immediate problem with real world content of establishing what the right make-up gain is for the volume lost to the compressor. Volume differences themselves sound like quality differences to regular folks.

Moreover, while a compressor certainly changes the signal, those changes are entirely unlike harmonic distortion, and I don't think you could apply THD measurement protocols to the output of a compressor. We've seen before folks thinking that because compression reduces peaks it must be the same as clipping peaks, but it's not. Look at it this way: a single band compressor is really, at its core, an automatic volume control. Everybody knows that if your amplifier is clipping you can turn it down to make the clipping go away. If that is the case, how could turning the volume down automatically BE clipping?

FWIW, once upon a time, wading through (IIRC) links from links found in the MJ Human sticky on receivers I found a couple of interesting things. One was testing performed to select the digital scheme to be used for digital broadcast radio in a Scandanavian country. Regular folks listened, voted, and a choice was made, then played for an expert in sonics and digital domain stuff, who immediately was able to identify an audible carrier signal (high pitched drone tone) always present in the selcted codec. Obviously low enough in level to go unnoticed by folks not acustomed to listening for it. Not a problem? Well, IMO, sooner or later some percentage of the population would, on their own, "discover" the carrier tone. Once they'd heard it they could never un-hear it and it could interfere with the ability to enjoy those radio broadcasts forever.

Another anecdote was a test to "prove" that op amps do not create audible distortions. The test as set up by op amp believers failed miserably because they had stacked op amps in a way that caused them not to behave linearly. IIRC, Every listener was able to identify the inserted op amps AS as different from the no-op-amp condition. Once the testers figured out they had bollixed up the setup, the redid the test with the op amps in their linear ranges and lo and behold, double blind testing found no differences.

DBT is a good thing. The interferences that come from being humans operate on a subconscious level so we cannot control them through acts of will however superhuman they may be. Of course, DBT demonstrates that Tylenol eases pain, but I'll take aspirin any day, because to me Tylenol seems not to be effective. Maybe variation among subjects, maybe some bias I picked up, but I'll pass on the Tylenol almost regardles of where or how painful the pain is.

Nick, to clarify the testing procedure detailed in the Meyer/Moran paper, they used at least one SACD/DVD-A universal player (not a CD player, per se, as you state above) to play only two channel high res media in several different playback systems. The universal player analog output was split -- one branch remained direct, while the other branch looped through a 16 bit 44.1 kHz A/D/A chain.

ABX testing of 554 trials produced 276 correct identifications, i.e. almost exactly chance for the entire testing group. At the individual level, one listener correctly identified 8/10 once, and two others correctly identified 7/10. If those listeners were tested only once each, they would warrant further trials to determine if their confidence levels remained the same. All other results were fewer than 7/10. The only scenario that allowed consistent identification of the redigitized signal was unnaturally elevated playback level with no audio input that then revealed the noise floor of the A/D/A chain.

If one were to pick nits with the Meyer/Moran testing procedure, the universal player and high res media selected would likely be the most ripe for scrutiny. Was the SACD/DVD-A player capable of greater than 16 bit 44.1 kHz equivalent performance? Were the high res media recorded and engineered to exceed 16 bit 44.1 kHz standards?

AJ

Sensitivity and Specificity

Thinking about this further, I think Nick's question actually goes to the question of "sensitivity and specificity." Here is the "wiki" on S&S:
http://en.wikipedia.org/wiki/Sensiti...nd_specificity

Sensitivity refers to a test's tendency to produce "false negatives". IOW, how often does the test show a negative result when the actual result is positive?

Specificity refers to a test's tendency to produce "false positives". IOW, how often does the test show a positive result when the actual result is negative?

I think Nick's question is primarily addressing the sensitivity of the blind ABX test... how often is there a real difference in sound quality that the ABX can not identify. Bottom line, I think this is a very legitimate question, and one for which I don't believe we have a good answer.

-----------------------------------------------------------------------------------------------------
EXAMPLE:
From the medical field, let me give an example:

When evaluating a patient for coronary artery disease, (CAD), the first test one would perform is an electrocardiogram, (ECG). The resting ECG is only about 20% to 70% sensitive for the diagnosis of coronary artery disease. IOW, 30 to 80% of patients who actually have coronary artery disease may not show that disease on a resting ECG. It is considered a fairly low sensitivity test.

If the patient doesn't have positive resting ECG findings for CAD, but the physician still suspects the patient has CAD, the next test used to confirm the diagnosis is the cardiac stress test. You put the patient on a treadmill and perform an ECG while they exercise. A cardiac stress test has a much higher sensitivity, along the order of 60 to 80%. Still, some 20 to 40% of patients can have CAD, and not show a positive result on stress testing.

The next test used to make the diagnosis is coronary angiography. A catheter is placed in the coronary artery, and radio-opaque dye is injected into the artery. X-rays are taken to visualize the inside of the coronary artery. This test is considered to be very highly sensitive. It is considered the "gold standard" for the diagnosis of CAD. The physician can actually *see* the disease and quantitative its' severity.
-----------------------------------------------------------------------------------------------------

This concept of sensitivity and specificity is used in every aspect of laboratory and clinical medicine. Every test is scrutinized for how often it produces false negatives and false positives. Obviously, you don't want to miss a diagnosis because of a false negative test. Just as importantly, you don't want to over-diagnose a medical condition and treat patients who don't actually have the condition based on a false positive test. You want a test that accurately diagnoses those patient who have the condition, while also accurately identifies those patients who don't have the condition. The ideal test has 100% sensitivity and 100% specificity. There are no such tests. Nonetheless, it is the goal.

To bring this back to ABX testing, I think Nick is questioning the sensitivity of the ABX test. (Specificity, (false positives), is not a part of his question.) IOW, Nick is asking, how often does this test produce a "false negative", where an actual difference exists, but the test is not sensitive enough to delineate it. This is a very real phenomenon in any kind of testing. I know of no documentation of the sensitivity of this test. If other forum members are aware of this information, please share it.

The lack of sensitivity/specificity information on the ABX test is yet another reason why I think it is inappropriate to extrapolate these test results. The combination of lack of sensitivity information, plus the limited sample size in this test, plus the "controversial" use of a switch, limits its applicability to broader generalizations about the audibility of these mods. There are simply too many unanswered questions to allow for the broader application of these results.

Craig