Observations of a controlled Cable Test

[ChrisWiggles]

[continued from http://www.avsforum.com/avs-vb/showt...18365&page=59]

Morning!

I got back a little late last night and didn't do a writeup then, I figure Mike will post his impressions at some point too.

First of all, kudos to Mike for being the always-gracious host, and also for his fairness and open mindedness for this trial. It is fair to recognize that he only had something to lose, and nothing really to be gained except perhaps an objective vindication of his position.

And to cut to the chase, Mike could not identify the Monster from the Opus MM with any accuracy (nor the reverse, which also would have been a positive result if he had been consistently wrong) using our testing methodology. We stopped the test a little less than halfway through, I think we got through 8 A/Bs before we gave up.

There were 4 of us present at the test. Mike had performed sighted listening tests on his own, and before I arrived he performed sighted listening tests while wearing a blindfold to acclimate himself to the testing method. His blindfold was a pair of duct-taped safety glasses. He could not see anything except a tiny bit directly below him. Theoretically, he could have strained his neck and been able to see the floor by the speakers where the wire was, but this was not allowed. After all, cheating would only serve to cheat himself. While switching wires, Mike wore earmuffs to minimize any possibility of identifying wires by the noises of switching the wires.

The plan was to do twenty A/B trials, which were chosen by flipping a coin. This was done before I arrived. After I arrived, we decided to check the level-match between wires. We did not have a method to level-match the wire if they were non-matched, however we also felt that the wire should match. Because the Opus uses some kind of network box, it conceivably would be possible that the signal were attenuated or otherwise altered and change the level, making it possible to identify the wire this way, rather than by sound 'quality' per se. The levels matched with a sine wave tone within 1/100th of a volt, the difference between the two sides was several hundredths of a volt with both wires. The difference between the L and R channels was greater than the possible difference between wires, which anyway was insignificant. We decided to check this because while we had no way of matching levels precisely, if there were a level difference we would have added a step to our methodology by turning down the volume all the way and then allowing mike to adjust the volume up, which would eliminate the ability to judge any difference in volume. We didn't need to do this, so Mike could listen without having to change the volume all the time, however he was free to change volume if he wanted.

There was, being generous, about one minute of a wait between changing cables. This was the case for both the sighted preparatory tests, and the blind tests. His amp basically goes straight through, so even with it turned off, it is feeding the speakers for a while. We would wait a little bit after turning off the amp before disconnecting the speaker wire and switching wire. We took care to be quiet while putting the wire down so as not to make any noise which could distinguish the wire. We changed the wire each and every time, regardless of whether switching to the other cable. At each time Mike would wear the earmuffs, even if he was going back to a known test, again to eliminate any possibility of identifying the wire by the noise of changing the wires.

We did at one point stop and go play pool for a few minutes and then come back to the test, doing a sighted (though still blindfolded) test to re-familiarize Mike with the cables.

So our results with Mike as our listener were clear: for this particular methodology, Mike could not accurately identify a difference in the cables. Again, a backwards result of Mike wrongly identifying the cables reliably also would have been a positive result.

I went into this with a fairly open mind. I did not have a previous opinion on the difference in speaker wire. However, I am fairly objective, and I did have two main expectations. First was that speaker wire, even if it did have an audible difference, would be very insignificant in the scope of the overall system. If there was going to be a difference, I was expecting it to be extremly subtle, and small. In other words, being objective about speaker wire tells me that it is fairly straightforward if you have sufficient gauge that the wire should basically get everything to the speaker with a precision that exceeds our hearing ability. Second, if there was going to be a difference, the suggested possibility of the network box in the Opus rolling off the highs may have caused an expectation of slightly less brightness/detail.

I personally went through the first few trials sighted, helping to change the wires. Going into the test I had no set opinion beside the expectations I just described. And while I am quite objective, objectivism also tells me things about vinyl which are quickly disproven in Mike's system. So I am open to these kinds of subjective things as well. As I went through 3 sighted trials, I began to form an opinion which basically told me that there WAS a difference in cables. Subtle, but I can say with confidence that it was present. I thought the Monster cable was a bit harsher, with a little bit less separation between instruments. This may have been a bias from my expectation that the Opus network box could have been attenuating the high frequencies, but I swore it was there. It is interesting to me that while I formed this opinion not having heard Mike or anyone else describe what they thought the difference was, after we stopped the whole test, Mike described what he previously thought the sighted difference sounded like (what he was listening for in the test) was practically identical to what I thought I was hearing sighted.

Then, for my own curiousity, I did the remaining 5 A/Bs before we stopped the test at 8 A/Bs blinded. I didn't wear ear muffs, and I just closed my eyes. I could have cheated if I wanted, but I did not (I guess you just have to take my word). I also, to check the rigor of our methdology, tried to actively listen to the cables being switched and see if I could identify either, to see if there was a way of identifying which was which that might have influenced mike even with earmuffs. There was no way to identify wire this way. I had looked at our test sheet, but I made no effort to try to memorize the sequence or have any idea what I would be listening to. In my 5 trials, I was certain on 3, and relatively uncertain on 2. I only matched 3 our ot 5, which is basically just blind luck, and I only got 2 of the 3 I was certain about correct. In other words, as a second, just self-interested and self-blinded tester, I could not with any reliability identify a difference in the cables.

The results for Mike and also the non rigorous one for myself certainly undid the opinion I was forming during my sighted listening. To characterize the position I've come to, I think it's fair to say that the objectivist arguments have a great deal of merit. It is fair to say that the onus is on subjectivists to discern speaker cable differences reliably. However it is ALSO fair to say that it is very difficult to prove a negative, especially for people with a subjectivist bent. Objectivists must also be honest about what our test results tell us: that with this methodology, and with this methodology ONLY, we could not reliably discern any differences in the cables. This does not prove that there are no differences with the cable, only that there were no differences discerned with this method.

In my personaly opinion, doing a long line of tests like this is generally unpleasant to do, you get tired of listening to the same stuff over and over, and after a little while it all starts to sound the same. Also, Mike only went back to a known A/B once during the test. I still would be interested in a similar test but spread out in time, doing maybe only one or two A/Bs at a time. This keeps fatigue down, and things don't all start to sound the same (again, just for the sake of argument assuming that there is an audible difference).

However, this test does tell us that even assuming for the sake of argument that there is a difference, that such a difference is extremly small, since it could not be percieved reliably with this method. After we finished the test, for curiosity we cut the ends of 16 gauge extension cords, and connected them to the system. We did this sighted. And the system sounded absolutely fantastic. The result I came away with through the whole experience is that people should generally not fret about speaker wire. Using a speaker wire of sufficient gauge for the task gets the job done. Even the unreliable sighted 'impressions' I had of the difference between Monster an Opus, which can not in any way be fairly attributable to actual difference, was extremely subtle. And the 16 gauge sounded basically exactly the same too.

After the test Mike was very gracious, and very fair and honest in accepting the result. I think he is probably curious to do some more testing in a different A/B format, perhaps along the lines I suggested. Or perhaps an ABX type test. It was curious that after the test, he characterized his certainty that during the test, it seemed fairly clear which was which. However, he was not identifying the cables with any consistency (rightly or wrongly) during the test. He did stop though and we left to go play pool to get away from the testing, so it seemed to me as an observer anyway that he was beginning to have difficulty, at least from his perspective, in identifying which was which. However, he was before that time not being at all consistent.

So I think that's basically everyting that either side would want to know. And I'm sure there's plenty of ammunition for both. I can say now that I am of the opinion that speaker wire basically makes no significant difference in sound quality, given sufficient gauge. Even the 16gauge extension cord was fantastic. I am still open to the possibility of audible differences with speaker cable, however such differences must be quite small and very subtle.

After the test, Mike played a handful of records, and I played some CDs as well. And what I came away with perhaps more than that is that Mike has probably the best overall system I've ever heard especially for BIG music. Big orchestral and rock sounded simply unbelievable. Santana, the Gates of Kiev, and Zeppelin among others on vinyl were just absolutely incredible. A completely new experience. I was trying to be extremely picky, and while I've heard a system here and there do small things differently in ways that in my opinion were slightly better, stepping back and considering the whole picture: it's simply an absolutely incredible system. The best I've ever heard big complex music.

So thanks again to Mike for being open and fair through the whole test and setting it all up and putting in the time and effort even though he really had nothing to gain. And for the pizza!

So flame on. If I were Mike, I frankly wouldn't care that much anyway, cable difference or not, it's one hell of a music system.

[Curt Palme]

I haven't kept up on this thread, or I probably would have found an excuse to come down..

An excellent writeup Chris! Your test pretty much covered how I would have done the a/bing, and the outcome was pretty much expected (coming from an objectivist like me! <grin>).

Now I'm REALLY curious as to Mike's comments. Mike, has this test changed your mind about cables, or ??

[speco2003]

well that certainly is a little surprise. Maybe it means that the network boxes are as big a myth as the 10 dollars woth of parts in the PSAudio noise harvester.

I find the extension cord test very telling as well.

And I would really like to hear records again on a good system so count that as a plus for you.

[AdrianMills]

Kudos all around to those involved.

[Swampfox]

Kudos to Mike and all those involved.
It be nice to hear Mike's impressions of the test.

[Dizzman]

Can all of those who are normally on an insulting and sarcastic bent please try to be adult.

The results here are (to my interpretation) far more about sighted biases than anything else.

And as will undoubtedly happen, they will be stated to be valid only for mikes system. "of course if they used system ABC, then it would have been clearly obvious"

Kudos to mike for stepping up. ANd Like Curt, i am most interested in mikes take on the test.

unless he is sitting in his barn weeping (joke, just kidding)

[AdrianMills]

Given the boxes on those cables I would not have been at all surprised if they were screwing with the signal enough to make an audible difference.

On the other hand I'm not at all surprised either that no difference could be heard but I'd feel more confident about that if an ABX switcher had been used.

[speco2003]

[quote=AdrianMills;12252132]Given the boxes on those cables I would not have been at all surprised if they were screwing with the signal enough to make an audible difference.
QUOTE]


Except they made no difference which probably means they are useless.

[Dizzman]

I made comments quite a while back on some other thread about this. When i talked to a friend of mine about this type of testing, he said that for unknown sounds, when comparing for subtle difference (i am REALLY paraphrasing here as this was a 2 hour convo) human auditory memory is about 4 seconds. i am talking a straight passage to passage comparison. Now the more knowledge i have (if i am a professional musician, then i can take those "tones" and put them into the context of what note they are, if the pitch is correct... etc) the longer that gap can be. but in general terms, our ability to tell if they are the same level or slightly different or such things, fades pretty quick after 4 seconds.

Not saying this as an excuse, but as another piece of info for the whole testing methodology thing. This is the reason why an ABX box is preferred, instantaneous switch.

[Gordon Shumway]

Well done...basically proved what I and many other friends of mine have known for a long long time...cables are pretty much cables...and it is indeed next to impossible to tell the differences between them when blindfolded. SO it all boils down to placebo effect I guess.

Sad to see so many folks spend a bunch of $$$ on cables thinking they are going to hear an "amazing difference" like the marketing guys who sell them like to spin it all.

Congrats to all who participated in helping dispell the cable myth...at least to some degree.

[Rutgar]

Well, I'm also surprised. However, I agree that everyone should try to remain adult about this. Also, we should wait to see what Mike has to say as well.

[Chu Gai]

Would be fun to run the MIT's through the same gauntlet don't you think, Rutgar?

[Dizzman]

Lets look into the test that was done

[Rutgar]

Originally Posted by Chu Gai

Would be fun to run the MIT's through the same gauntlet don't you think, Rutgar?

I'm not opposed to doing that. But at this point lets wait until Mike chimes in, and let's let the dust settle on this test before everyone start clamoring for another.

[ChrisWiggles]

Originally Posted by Gordon Shumway

Well done...basically proved what I and many other friends of mine have known for a long long time...cables are pretty much cables...and it is indeed next to impossible to tell the differences between them when blindfolded. SO it all boils down to placebo effect I guess.

I think it is important to point out that this is not what the test proved affirmatively. While it's buried in my text, it is very important to be clear that this test proves only one thing: that with this particular testing methodology, it was not possible to distinguish any difference between these cables. It does not prove that there are no audible differences between cables or even between these two specific cables. Just that there weren't audible differences the way we tested. It is possible, though in my personal opinion rather unlikely, that a different methodology could yield different results. So for subjectivists who are certain that there are differences in cables, our test does not authoritatively disprove that opinion, however it does shed some doubt as to that position at least on the degree of difference that some subjectivists claim. Clearly, there was not any degree of difference large enough to be discerned with our test, which means that there could be a degree of difference below the perceptual threshold allowed by our test, that the degree of difference is fully below our audible threshold for all possible methodologies, or that there is no difference whatsoever. For objectivists who already were of the opinion that the cables couldn't have an audible effect, this test taken alone bolsters, but does not authoritatively vindicate that position.

It is far more difficult to prove that there isn't a difference. Given the statements by many subjectivists that the difference is easy to discern, it should be easier to prove that there is a difference. I think that all this test shows us is that, even assuming for the sake of argument that there are audible differences, that such differences are much harder to discern than many may claim. But this test taken alone does NOT prove that there are no audible differences in cables. Objectivists who arrive at that position do so based on the merits of other knowledge about both cables, and our hearing capabilities. You cannot arrive at that conclusion based only on this test, which was only capable of determining that under these specific methods and with these specific cables, there were no audible differences.

We just have to remain honest about both the clarity of the result and the limits of applicability of that result. This test alone is not close to universally definitive for these reasons.

[Hughman]

Great write-up Chris and thanks to Mike L. taking one if you know what I mean.

The results do tend to lend creedence to the cables sound the same group but I'm not sure a scientist would draw any general sweeping tidy conclusions at this point at least not from this test alone. The results suggest that this Monster cable used can not audibly be differentiated from the Opus during blind-folded testing with delay of about one minute between tests. The next set of tests or research should now concentrate on auditory memory to determine how that factor could have influenced the outcome. Perhaps all this test proves is poor audio memory at one minute test intervals.

Another area of concern is the blind-folded portion of the listening, often when I implement system changes it's the differences in imaging that is most telling, sound changes constantly for me depending on many factors BUT imaging of a particular song stays relatively constant, predictable and repeatable. Ignoring or hobbling this important area of this hobby by blind-folding the participants and thereby losing image location references is akin to blindfolded testing for color differentiation. Perhaps imaging is a placebo effect at play, would be interesting to perform some tests on that aspect of audio.

[ChrisWiggles]

Originally Posted by Hughman

Great write-up Chris and thanks to Mike L. taking one if you know what I mean.

The results do tend to lend creedence to the cables sound the same group but I'm not sure a scientist would draw any general sweeping tidy conclusions at this point at least not from this test alone. The results suggest that this Monster cable used can not audibly be differentiated from the Opus during blind-folded testing with delay of about one minute between tests. The next set of tests or research should now concentrate on auditory memory to determine how that factor could have influenced the outcome. Perhaps all this test proves is poor audio memory at one minute test intervals.

That is absolutely correct, and a fair characterization.

However, it did also prove that the difference percieved using a sighted test, with the same test interval, was largely in our heads. That is also a very interesting aspect of the result, as I also was certain that I was hearing a subtle difference between cables, and this is coming from someone who despite my explicit desire to be open, impartial, and non-opinionated, am naturally something of an objectivist. There are certain aspects of my expectations that I cannot simply toss aside, such as my expectation that speaker cables, even assuming an audible difference, would pale in significance to the impact of room, speakers, electronics. And yet I too was quite convinced that there was a subtle difference during the sighted trials. This is an interesting phenomenon. It was what prompted me to try my own hand at seeing how difficult it was to distinguish between the two cables, partly because I thought that given my impressions sighted, I could do it reliably blinded. Part of that motivation was that Mike got the second two wrong, and I thought that perhaps I could do it better, that surely given what I was hearing sighted, I could differentiate between the to. Well, we both failed equally!

Another area of concern is the blind-folded portion of the listening, often when I implement system changes it's the differences in imaging that is most telling, sound changes constantly for me depending on many factors BUT imaging of a particular song stays relatively constant, predictable and repeatable. Ignoring or hobbling this important area of this hobby by blind-folding the participants and thereby losing image location references is akin to blindfolded testing for color differentiation. Perhaps imaging is a placebo effect at play, would be interesting to perform some tests on that aspect of audio.

That's possible. I personally am of the opinion that not seeing the speakers improves imaging, so I think that being blindfolded actually helps hear the system better. In addition, eliminating senses is a classic way of focusing and sharpening our other remaining senses. So if anything, I think all of us present at the test were under the assumption that being blindfolded would aid, rather than inhibit the test. Certainly if one wanted to avoid being blindfolded, you could rig up a screen at the speakers so you could not see behind them, if that was your preference.

[Hughman]

Originally Posted by ChrisWiggles

That's possible. I personally am of the opinion that not seeing the speakers improves imaging, so I think that being blindfolded actually helps hear the system better. In addition, eliminating senses is a classic way of focusing and sharpening our other remaining senses. So if anything, I think all of us present at the test were under the assumption that being blindfolded would aid, rather than inhibit the test. Certainly if one wanted to avoid being blindfolded, you could rig up a screen at the speakers so you could not see behind them, if that was your preference.

I would think it would be very difficult to accurately remember imaging without a sighted reference point ie: that bell eminates exactly 1 foot out from that little spot on the wall. Much of the difficulty of proving or disproving any of this is out memory and the noisefloor this creates possibly obscuring changes. Using a reference point to place an image can pretty much remove this memory factor from the imaging portion of the test. But as you say, blind-folding may very well aid in the auditory portion of the test.

Terrific testing though and what an eye opener it must have been.

[krabapple]

Originally Posted by ChrisWiggles

I think it is important to point out that this is not what the test proved affirmatively. While it's buried in my text, it is very important to be clear that this test proves only one thing: that with this particular testing methodology, it was not possible to distinguish any difference between these cables.

It didn't actually prove that either to a high degree of probability. You only did 8 trials! So 'impossibility' was hardly demonstrated.

But it does have two important facets:

1) Mike was pretty confident he heard differences; the stats belie that confidence. Chris W also 'confidently' heard differences sighted, that had no support when 'blinded'.

2) thus pointing up HOW VERY EASY IT IS for people to convince themselves, 'confidently' that differences exist...in sighted comparison. When in fact this confidence may not be warranted in the least. To me this is key -- the psychological fact is, people can and do tend to underestimate how easily their senses can be fooled.


Chris, I also wonder about this statement of yours

objectivism also tells me things about vinyl which are quickly disproven in Mike's system.

What did 'objectivism; tell you about vinyl that was quickly disproven in Mike's system? Objectivism doesn't 'tell' us that LP's can't sound great...or even that they MUST sound subjectively worse than a CD of the same material.

As for anyone's objections about switching, etc...everyone needs to keep in mind that Mike did his OWN sighted tests, and also sighted trials during this test, where he was CONFIDENT he heard differences, under 'slow switch', nonABX conditions. *THAT* is the phenomenon that was tested; indeed, that is the phenomenon he kept reporting for his cables, the phenomenon that got him interested in Randi's challenge in the first place. Not some 'possible' difference that might be heard under more discriminative conditions. He was being tested on differences he already claimed to hear.

[ChrisWiggles]

Originally Posted by krabapple

It didn't actually prove that either to a high degree of probability. You only did 8 trials! So 'impossibility' was hardly demonstrated.

It most certainly did. The result over those 8 trials, and my 5 was just statistically random guessing. We stopped at 8 because it was quite clear that it was just random. I believe our requirement for a positive result was either 10 out of 10, or 17 out of 20, we intended to do 20 ABs. Less than that and it could have been criticized as just good luck.

[quote]

What did 'objectivism; tell you about vinyl that was quickly disproven in Mike's system? Objectivism doesn't 'tell' us that LP's can't sound great.

Certainly. But there is a lot of data that tells us that there is less detail in LP compared to CD. True or not, what I heard on Mike's system, scratches, high noise floor and all, was simply incredible. It wasn't by any means scientific, I just forwarded that as evidence that I approached the trial with an open mind. I was perfectly open to hearing cable differences, and indeed during the sighted trials I was confident that I was. It would be very easy for me to enter the trial with a great deal of bias if I so chose, dead-set against hearing any differences, convinced it was simply impossible. I tried my very best to go in with a completely open mind, having formed no opinion on the matter. That is never attainable with anything, but objectivity also involves impartiality or its pursuit. I don't think subjectivists can simply dismiss my presence or my own interpretations as being pre-emptively biased, and the same goes for everyone else there. These kinds of things, as always, should always be learning experiences, rather than ideological pursuits. And I think that is part of being objectively fair and open minded as well. If Mike had reliably heard the differences blinded, I would be just as open to that possibility.

[krabapple]

Originally Posted by ChrisWiggles

It most certainly did. The result over those 8 trials, and my 5 was just statistically random guessing. We stopped at 8 because it was quite clear that it was just random. I believe our requirement for a positive result was either 10 out of 10, or 17 out of 20, we intended to do 20 ABs. Less than that and it could have been criticized as just good luck.

Well, that's *your* requirement. But one could propose that the*real* difference is a very subtle difference, requiring lots of trials. In any case, you used careful terminology everywhere in your post except here -- as you said, you can't prove a negative, so you didn't prove that the difference was *not possible*. Why not just say that "the hypothesis of difference was not supported at x confidence level", as a scientist would, and leave it at that?

What, btw was the score correct out of 8 trials?

Certainly. But there is a lot of data that tells us that there is less detail in LP compared to CD.

Again, this is vague wording. What is the criterion for 'detail' you have in mind. There's objective data too that LP imparts an illusion of dynamic range that it doesn't actually have; that it imparts 'imaging' and 'ambience' impressions that are in part actually due to distortion -- on other words, there are data about LP having euphonic distortion characteristics. Subjectively one might interpret one or more of these as 'more detail than I expected'...or not.

For more, read James Johnston's (JJs) posts on the 'CD Vs vinyl deathmatch' thread

True or not, what I heard on Mike's system, scratches, high noise floor and all, was simply incredible.

Well, that's great -- I've heard terrific LP playback too, most recently at a home expo with top-end gear -- but subjectively excellent turntable playback is not contrary to science, nor is it impossible that an LP can sound better than a CD -- and even 'measure better' in some parameters (e.g., comparing a CD mastering with highly restricted dynamic range to its LP counterpart with a wider DR). The reason to be amazed would be if you had a bias towards thinking it COULDN'T EVER sound good.

Unscientifically, I'd say anyone who's spent as much on his TT playback chain as Mike, had BETTER have excellent sound as a result.

[Michael Grant]

First of all let me echo my thanks to Mike and Chris for doing this test, and to the unnamed participants who helped.

krabapple's careful parsing is right. All this proved was that Mike failed one test. It does not prove that Mike could not ultimately distinguish between the cables, even under these specific conditions. Perhaps given more opportunity to become comfortable with the testing conditions, the result might be different. I don't believe that, of course, but I do think it's important to give the right interpretation to the test results.

Also, the more trials that one does, the smaller the deviation you need from 50% to get firm confidence in a non-random element. So by stopping the test early you cannot conclude that the results are the same as random. In short, the more trials in advance you decide to do, the more times you get to fail before you can call the test off as a failure.

I do think this test provides a clear example of the self-deception that is possible when tests are not properly blinded. If I read Chris right, Mike was able to "practice" with the test conditions in a non-blind manner. Presumably he was able to distinguish the cables under these "nearly controlled" conditions, or he wouldn't have bothered to proceed with the blind tests. I think this is a very useful component of any test of this nature---the ability to run through it "non-blind". After all, you do want to do your best to make sure that the testing methodology itself isn't unduly compromising the results.

I hope that Mike and Chris are encouraged by this test to consider other approaches. For instance, I would really like them to consider a test with some sort of ABX switch box in place. I know there is concern that the added circuitry could compromise the test---but again, that's why it is important to do the non-blind pre-test first. With an ABX box this is easy, because you can in fact know what "A" and "B" are even in the controlled setting. It's only "X" that is concealed. So, if you believe you can distinguish between A and B with the box in place, then the full ABX can proceed with confidence.

The point is that no test is perfect. In this case, the slow switching times were a factor; in the case of an ABX box, the circuit itself might be a factor. But by approaching the question from several angles you can conceivably get a confident result. The only thing you cannot do, of course, is remove the blinds.

[CharlesJ]

Yes, absolutely, Kudos to all. No small task to have done this.

Originally Posted by speco2003

Maybe it means that the network boxes are as big a myth as the 10 dollars woth of parts in the PSAudio noise harvester.

I find the extension cord test very telling as well.

Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.

Yes, that 16ga was an interesting add on. But at such a short distance no contest. Similar results what the Greenberg tests from the 1990s.

[Rutgar]

Originally Posted by Michael Grant

I hope that Mike and Chris are encouraged by this test to consider other approaches. For instance, I would really like them to consider a test with some sort of ABX switch box in place. I know there is concern that the added circuitry could compromise the test---but again, that's why it is important to do the non-blind pre-test first. With an ABX box this is easy, because you can in fact know what "A" and "B" are even in the controlled setting. It's only "X" that is concealed. So, if you believe you can distinguish between A and B with the box in place, then the full ABX can proceed with confidence.

The point is that no test is perfect. In this case, the slow switching times were a factor; in the case of an ABX box, the circuit itself might be a factor. But by approaching the question from several angles you can conceivably get a confident result. The only thing you cannot do, of course, is remove the blinds.

Michael, I agree that it might be better to try it again with some sort of ABX box setup. As Chris indicated, the time in between the listening could have skewed the test. Especially after a lot of listening of the same thing over and over. I found this to be a problem when I was at the Rocky Mountain Audio Festival last month. After going from room to room to room to room... I got tired of 'listening'. We had our own demo disc with us, and played it in most of the rooms we visited. I got very, VERY sick of that demo disk. And after a while, it pretty much sounded the same on almost everything we heard.

[ChrisWiggles]

[quote=krabapple;12253444]Well, that's *your* requirement. But one could propose that the*real* difference is a very subtle difference, requiring lots of trials.[quote]

The point of using more trials doesn't have anything to do with the degree of difference, but rather in making sure that the result is not reasonably possible simply at random. Assuming for a moment that there is no difference whatsoever, and each is a guess, the statistical probability of guessing correctly over the number of trials has to be low enough that it is extremely unlikely that we could get a wrong result simply by freak luck. If we did 5 out of 5, that's 1 out of 32 chance that someone could simply guess randomly and match each. What if Mike had matched 4 out of 5? That's quite attractive for someone to say "ahah! there is a noticeable difference" but that's not a very strong result. The possibility of matching 10 out of 10, or 17 out of 20 from pure chance correctly is extremely low.

In any case, you used careful terminology everywhere in your post except here -- as you said, you can't prove a negative, so you didn't prove that the difference was *not possible*. Why not just say that "the hypothesis of difference was not supported at x confidence level", as a scientist would, and leave it at that?

I thought I had made it very clear. We didn't by any means prove that there was no difference, audible or otherwise. We simply illustrated that using this testing methodology, we could not identify which cable was which with any degree of accuracy. The results suggest that it was simply random guessing, whether Mike or myself thought we were guessing or not.

What, btw was the score correct out of 8 trials?

You know, that's a good question, and actually I forget. That's somewhat embarrassing I know.

Again, this is vague wording. What is the criterion for 'detail' you have in mind. There's objective data too that LP imparts an illusion of dynamic range that it doesn't actually have; that it imparts 'imaging' and 'ambience' impressions that are in part actually due to distortion -- on other words, there are data about LP having euphonic distortion characteristics. Subjectively one might interpret one or more of these as 'more detail than I expected'...or not.

For more, read James Johnston's (JJs) posts on the 'CD Vs vinyl deathmatch' thread

This is sort of an off-topic discussion. I'm not hugely informed about vinyl. All I know is that I've heard Santana Gypsy Queen a million times, and never in my life has it been like that.

My statement about TT was just an aside example to illustrate that I'm open to subjective experiences that run contrary to my objective understanding. Perhaps my objective understanding about vinyl is wrong, but I was of the impression that it was fairly clear-cut that CD is plenty capable of keeping up technically, or exceeding vinyl. Whether that is right or not is beside the point. Rather, I objectively approached the issue, but I am open to unexpected experiences. Otherwise, I might just as easily reject what I thought I heard, and that's not fair to do either.

[Michael Grant]

We simply illustrated that using this testing methodology, we could not identify which cable was which with any degree of accuracy. The results suggest that it was simply random guessing, whether Mike or myself thought we were guessing or not.

Yes, given this testing methodology including the pre-agreed 20 trial limit I think this is accurate.

The point of using more trials doesn't have anything to do with the degree of difference,

In a way, it does have something to do with this. If indeed the differences are subtle then it might not be possible to make a proper guess every time. Therefore, in order to "accept" a fair number of errors you have to do a lot of trials.

Let's suppose, for instance, that you can correctly guess the exotic cable 55% of the time. That may not seem like much, but since it's greater than 50% it is still a "win." On the other hand, since it is so close to 50%, you have to do a lot of trials to establish it.

[ChrisWiggles]

Originally Posted by Michael Grant

Yes, given this testing methodology including the pre-agreed 20 trial limit I think this is accurate.In a way, it does have something to do with this. If indeed the differences are subtle then it might not be possible to make a proper guess every time. Therefore, in order to "accept" a fair number of errors you have to do a lot of trials.

Let's suppose, for instance, that you can correctly guess the exotic cable 55% of the time. That may not seem like much, but since it's greater than 50% it is still a "win." On the other hand, since it is so close to 50%, you have to do a lot of trials to establish it.

That's why you're a mathematician. You're right.

[speco2003]

[quote=CharlesJ;12253568]

Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.
QUOTE]

Perhaps they do but I "think" on the website they state audible differences which clearly just are not there. I have to go out and mix the 7 o clock show so I will have to check the Opus website in a couple of hours.

[krabapple]

Originally Posted by Michael Grant

First of all let me echo my thanks to Mike and Chris for doing this test, and to the unnamed participants who helped.

krabapple's careful parsing is right. All this proved was that Mike failed one test. It does not prove that Mike could not ultimately distinguish between the cables, even under these specific conditions. Perhaps given more opportunity to become comfortable with the testing conditions, the result might be different. I don't believe that, of course, but I do think it's important to give the right interpretation to the test results.

Indeed. (And btw kudos to Mike, Chris and co for doing the test at all). My understanding is that one should agree on a set number of trials beforehand and NEVER stop before the agreed-upon number of trials have been done, unless the subject needs to stop , because of fatigue or he/she no longer believes he/she hears any differences...i.e., the subject think's it's guessing.

To illustrate the danger, imagine they'd stopped after Mike got 6/8 correct when they'd agreed to do 20 trials beforehand; all sorts of alarms could be legitimately raised about cherry-picked results (6/8 is no guarantee that X/20 will remain significantly different from chance). It's really no different for negative results....*unless* the reasons I mentioned hold. And even then, they should try to finish the test at a later time, assuming the subject 'regains' his ability to hear 'differences'.

It actually would have been best to have Mike's answers be UNKNOWN to the proctors during the test, until all the agreed-upon trials were done, so that there's no temptation to stop it early.

I do think this test provides a clear example of the self-deception that is possible when tests are not properly blinded. If I read Chris right, Mike was able to "practice" with the test conditions in a non-blind manner. Presumably he was able to distinguish the cables under these "nearly controlled" conditions, or he wouldn't have bothered to proceed with the blind tests. I think this is a very useful component of any test of this nature---the ability to run through it "non-blind". After all, you do want to do your best to make sure that the testing methodology itself isn't unduly compromising the results.

Indeed again... a self-confessed inability to distinguish A from B while 'sighted', is a reason for not proceeding.

[krabapple]

Originally Posted by CharlesJ

Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.

I would say that no competently-built amp requires an outboard RF filter, but it's a fact that some 'high end' designs DO, and in those bizarre cases a cable with an RF filter has an advantage over one that doesn't...an 'unnatural' advantage, I'd say.