AVS Forum banner

21 - 40 of 384 Posts

·
Registered
Joined
·
5,788 Posts

Quote:
Originally Posted by ChrisWiggles /forum/post/12253334


It most certainly did. The result over those 8 trials, and my 5 was just statistically random guessing. We stopped at 8 because it was quite clear that it was just random. I believe our requirement for a positive result was either 10 out of 10, or 17 out of 20, we intended to do 20 ABs. Less than that and it could have been criticized as just good luck.


Well, that's *your* requirement. But one could propose that the*real* difference is a very subtle difference, requiring lots of trials. In any case, you used careful terminology everywhere in your post except here -- as you said, you can't prove a negative, so you didn't prove that the difference was *not possible*. Why not just say that "the hypothesis of difference was not supported at x confidence level", as a scientist would, and leave it at that?


What, btw was the score correct out of 8 trials?

Quote:
Certainly. But there is a lot of data that tells us that there is less detail in LP compared to CD.

Again, this is vague wording. What is the criterion for 'detail' you have in mind. There's objective data too that LP imparts an illusion of dynamic range that it doesn't actually have; that it imparts 'imaging' and 'ambience' impressions that are in part actually due to distortion -- on other words, there are data about LP having euphonic distortion characteristics. Subjectively one might interpret one or more of these as 'more detail than I expected'...or not.


For more, read James Johnston's (JJs) posts on the 'CD Vs vinyl deathmatch' thread

Quote:
True or not, what I heard on Mike's system, scratches, high noise floor and all, was simply incredible.

Well, that's great -- I've heard terrific LP playback too, most recently at a home expo with top-end gear -- but subjectively excellent turntable playback is not contrary to science, nor is it impossible that an LP can sound better than a CD -- and even 'measure better' in some parameters (e.g., comparing a CD mastering with highly restricted dynamic range to its LP counterpart with a wider DR). The reason to be amazed would be if you had a bias towards thinking it COULDN'T EVER sound good.


Unscientifically, I'd say anyone who's spent as much on his TT playback chain as Mike, had BETTER have excellent sound as a result.
 

·
Registered
Joined
·
10,201 Posts
First of all let me echo my thanks to Mike and Chris for doing this test, and to the unnamed participants who helped.


krabapple's careful parsing is right. All this proved was that Mike failed one test. It does not prove that Mike could not ultimately distinguish between the cables, even under these specific conditions. Perhaps given more opportunity to become comfortable with the testing conditions, the result might be different. I don't believe that, of course, but I do think it's important to give the right interpretation to the test results.


Also, the more trials that one does, the smaller the deviation you need from 50% to get firm confidence in a non-random element. So by stopping the test early you cannot conclude that the results are the same as random. In short, the more trials in advance you decide to do, the more times you get to fail before you can call the test off as a failure.


I do think this test provides a clear example of the self-deception that is possible when tests are not properly blinded. If I read Chris right, Mike was able to "practice" with the test conditions in a non-blind manner. Presumably he was able to distinguish the cables under these "nearly controlled" conditions, or he wouldn't have bothered to proceed with the blind tests. I think this is a very useful component of any test of this nature---the ability to run through it "non-blind". After all, you do want to do your best to make sure that the testing methodology itself isn't unduly compromising the results.


I hope that Mike and Chris are encouraged by this test to consider other approaches. For instance, I would really like them to consider a test with some sort of ABX switch box in place. I know there is concern that the added circuitry could compromise the test---but again, that's why it is important to do the non-blind pre-test first. With an ABX box this is easy, because you can in fact know what "A" and "B" are even in the controlled setting. It's only "X" that is concealed. So, if you believe you can distinguish between A and B with the box in place, then the full ABX can proceed with confidence.


The point is that no test is perfect. In this case, the slow switching times were a factor; in the case of an ABX box, the circuit itself might be a factor. But by approaching the question from several angles you can conceivably get a confident result. The only thing you cannot do, of course, is remove the blinds.
 

·
Registered
Joined
·
5,094 Posts
Yes, absolutely, Kudos to all. No small task to have done this.


Quote:
Originally Posted by speco2003 /forum/post/12251456


Maybe it means that the network boxes are as big a myth as the 10 dollars woth of parts in the PSAudio noise harvester.


I find the extension cord test very telling as well.

Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.


Yes, that 16ga was an interesting add on. But at such a short distance no contest. Similar results what the Greenberg tests from the 1990s.
 

·
Registered
Joined
·
5,399 Posts

Quote:
Originally Posted by Michael Grant /forum/post/12253472



I hope that Mike and Chris are encouraged by this test to consider other approaches. For instance, I would really like them to consider a test with some sort of ABX switch box in place. I know there is concern that the added circuitry could compromise the test---but again, that's why it is important to do the non-blind pre-test first. With an ABX box this is easy, because you can in fact know what "A" and "B" are even in the controlled setting. It's only "X" that is concealed. So, if you believe you can distinguish between A and B with the box in place, then the full ABX can proceed with confidence.


The point is that no test is perfect. In this case, the slow switching times were a factor; in the case of an ABX box, the circuit itself might be a factor. But by approaching the question from several angles you can conceivably get a confident result. The only thing you cannot do, of course, is remove the blinds.

Michael, I agree that it might be better to try it again with some sort of ABX box setup. As Chris indicated, the time in between the listening could have skewed the test. Especially after a lot of listening of the same thing over and over. I found this to be a problem when I was at the Rocky Mountain Audio Festival last month. After going from room to room to room to room... I got tired of 'listening'. We had our own demo disc with us, and played it in most of the rooms we visited. I got very, VERY sick of that demo disk. And after a while, it pretty much sounded the same on almost everything we heard.
 

·
Banned
Joined
·
20,735 Posts
Discussion Starter #25

Well, that's *your* requirement. But one could propose that the*real* difference is a very subtle difference, requiring lots of trials.
The point of using more trials doesn't have anything to do with the degree of difference, but rather in making sure that the result is not reasonably possible simply at random. Assuming for a moment that there is no difference whatsoever, and each is a guess, the statistical probability of guessing correctly over the number of trials has to be low enough that it is extremely unlikely that we could get a wrong result simply by freak luck. If we did 5 out of 5, that's 1 out of 32 chance that someone could simply guess randomly and match each. What if Mike had matched 4 out of 5? That's quite attractive for someone to say "ahah! there is a noticeable difference" but that's not a very strong result. The possibility of matching 10 out of 10, or 17 out of 20 from pure chance correctly is extremely low.

Quote:
In any case, you used careful terminology everywhere in your post except here -- as you said, you can't prove a negative, so you didn't prove that the difference was *not possible*. Why not just say that "the hypothesis of difference was not supported at x confidence level", as a scientist would, and leave it at that?

I thought I had made it very clear. We didn't by any means prove that there was no difference, audible or otherwise. We simply illustrated that using this testing methodology, we could not identify which cable was which with any degree of accuracy. The results suggest that it was simply random guessing, whether Mike or myself thought we were guessing or not.

Quote:
What, btw was the score correct out of 8 trials?

You know, that's a good question, and actually I forget. That's somewhat embarrassing I know.


Quote:
Again, this is vague wording. What is the criterion for 'detail' you have in mind. There's objective data too that LP imparts an illusion of dynamic range that it doesn't actually have; that it imparts 'imaging' and 'ambience' impressions that are in part actually due to distortion -- on other words, there are data about LP having euphonic distortion characteristics. Subjectively one might interpret one or more of these as 'more detail than I expected'...or not.


For more, read James Johnston's (JJs) posts on the 'CD Vs vinyl deathmatch' thread

This is sort of an off-topic discussion. I'm not hugely informed about vinyl. All I know is that I've heard Santana Gypsy Queen a million times, and never in my life has it been like that.


My statement about TT was just an aside example to illustrate that I'm open to subjective experiences that run contrary to my objective understanding. Perhaps my objective understanding about vinyl is wrong, but I was of the impression that it was fairly clear-cut that CD is plenty capable of keeping up technically, or exceeding vinyl. Whether that is right or not is beside the point. Rather, I objectively approached the issue, but I am open to unexpected experiences. Otherwise, I might just as easily reject what I thought I heard, and that's not fair to do either.
 

·
Registered
Joined
·
10,201 Posts

Quote:
We simply illustrated that using this testing methodology, we could not identify which cable was which with any degree of accuracy. The results suggest that it was simply random guessing, whether Mike or myself thought we were guessing or not.

Yes, given this testing methodology including the pre-agreed 20 trial limit I think this is accurate.
Quote:
The point of using more trials doesn't have anything to do with the degree of difference,

In a way, it does have something to do with this. If indeed the differences are subtle then it might not be possible to make a proper guess every time. Therefore, in order to "accept" a fair number of errors you have to do a lot of trials.


Let's suppose, for instance, that you can correctly guess the exotic cable 55% of the time. That may not seem like much, but since it's greater than 50% it is still a "win." On the other hand, since it is so close to 50%, you have to do a lot of trials to establish it.
 

·
Banned
Joined
·
20,735 Posts
Discussion Starter #27

Quote:
Originally Posted by Michael Grant /forum/post/12254034


Yes, given this testing methodology including the pre-agreed 20 trial limit I think this is accurate.In a way, it does have something to do with this. If indeed the differences are subtle then it might not be possible to make a proper guess every time. Therefore, in order to "accept" a fair number of errors you have to do a lot of trials.


Let's suppose, for instance, that you can correctly guess the exotic cable 55% of the time. That may not seem like much, but since it's greater than 50% it is still a "win." On the other hand, since it is so close to 50%, you have to do a lot of trials to establish it.

That's why you're a mathematician.
You're right.
 

·
Registered
Joined
·
2,365 Posts

Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.

QUOTE]


Perhaps they do but I "think" on the website they state audible differences which clearly just are not there. I have to go out and mix the 7 o clock show so I will have to check the Opus website in a couple of hours.
 

·
Registered
Joined
·
5,788 Posts

Quote:
Originally Posted by Michael Grant /forum/post/12253472


First of all let me echo my thanks to Mike and Chris for doing this test, and to the unnamed participants who helped.


krabapple's careful parsing is right. All this proved was that Mike failed one test. It does not prove that Mike could not ultimately distinguish between the cables, even under these specific conditions. Perhaps given more opportunity to become comfortable with the testing conditions, the result might be different. I don't believe that, of course, but I do think it's important to give the right interpretation to the test results.

Indeed. (And btw kudos to Mike, Chris and co for doing the test at all). My understanding is that one should agree on a set number of trials beforehand and NEVER stop before the agreed-upon number of trials have been done, unless the subject needs to stop , because of fatigue or he/she no longer believes he/she hears any differences...i.e., the subject think's it's guessing.


To illustrate the danger, imagine they'd stopped after Mike got 6/8 correct when they'd agreed to do 20 trials beforehand; all sorts of alarms could be legitimately raised about cherry-picked results (6/8 is no guarantee that X/20 will remain significantly different from chance). It's really no different for negative results....*unless* the reasons I mentioned hold. And even then, they should try to finish the test at a later time, assuming the subject 'regains' his ability to hear 'differences'.


It actually would have been best to have Mike's answers be UNKNOWN to the proctors during the test, until all the agreed-upon trials were done, so that there's no temptation to stop it early.


Quote:
I do think this test provides a clear example of the self-deception that is possible when tests are not properly blinded. If I read Chris right, Mike was able to "practice" with the test conditions in a non-blind manner. Presumably he was able to distinguish the cables under these "nearly controlled" conditions, or he wouldn't have bothered to proceed with the blind tests. I think this is a very useful component of any test of this nature---the ability to run through it "non-blind". After all, you do want to do your best to make sure that the testing methodology itself isn't unduly compromising the results.


Indeed again... a self-confessed inability to distinguish A from B while 'sighted', is a reason for not proceeding.
 

·
Registered
Joined
·
5,788 Posts

Quote:
Originally Posted by CharlesJ /forum/post/12253568


Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.

I would say that no competently-built amp requires an outboard RF filter, but it's a fact that some 'high end' designs DO, and in those bizarre cases a cable with an RF filter has an advantage over one that doesn't...an 'unnatural' advantage, I'd say.
 

·
Registered
Joined
·
5,788 Posts

Quote:
Originally Posted by ChrisWiggles /forum/post/12254087


That's why you're a mathematician.
You're right.

He sure saved me a lot of typing.



But here's another discussion (two posts) of the interaction of expected size of effect, p value, and number of trials (with a lot more typing...by someone else). It's also a good precis on good audio DBT practices:



"What is a Blind ABX Test?"
http://www.hydrogenaudio.org/forums/...howtopic=16295


and here's a link to a useful excel table of binomial probabilities of guessing

http://www.kikeg.arrakis.es/winabx/bino_dist.zip


basically from this you can read the p value that the result's due to chance, for any combination of #right/#trials up to 100/100.


For 8 trials, the p-values are shown below (from 1 to 8 correct). Mike would have to get 7/8 to be in the ballpark of standard scientific acceptability (p
 

·
Registered
Joined
·
15,247 Posts
Well, Mike can now auction off his specially prepared Monster cables - the ones that sound so close to the Transparents, you can't tell them apart.
 

·
Registered
Joined
·
251 Posts

Quote:
Originally Posted by Chu Gai /forum/post/12254694


Well, Mike can now auction off his specially prepared Monster cables - the ones that sound so close to the Transparents, you can't tell them apart.

Yes, must be worth 30K at least...



Damn, but at those prices I'd want to hear a difference not only blind but blind drunk too.


It just sunk in that for the price of a pair of 25' Transparent Opus cables (43k according to gizmodo) you can buy (in the US) a 5.2 system made up of 5 x B&W 802D and a couple of JLAudio f113 and still have change.
Holy cow. No, really.
 

·
Registered
Joined
·
5,094 Posts

Quote:
Originally Posted by speco2003 /forum/post/12254304

Quote:
Originally Posted by CharlesJ /forum/post/12253568



Perhaps the boxes do what the patents are for, RF noise reduction, not audio band shaping.

Perhaps they do but I "think" on the website they state audible differences which clearly just are not there. I have to go out and mix the 7 o clock show so I will have to check the Opus website in a couple of hours.

Yes, exactly. So the common notion that the box may or will cause an audible difference is not founded in anything and the higher probability is that the box is for RF filtering and not audio band alterations.

As to their web site claiming the difference, that is par for the course no matter what they place on the wire, or just having plain wires.
But, it could very well be that is where the common notion came from, jumping to an unfounded conclusion.


As posted elsewhere, someone placed this cable or somthing similar on a spectrum analizer and didn't see the box doing anything in the audio band.
I just don't have a link to that message
 

·
Registered
Joined
·
5,094 Posts

Quote:
Originally Posted by Hughman /forum/post/12252872


The results suggest that this Monster cable used can not audibly be differentiated from the Opus during blind-folded testing with delay of about one minute between tests. .

Yes, but that doesn't stop hearing differences under sighted conditions with 1 minute or more of setup time


But you are correct, acoustic memory has been shown to be very short, on the order of seconds. So, even long segments of music is detrimental.
 

·
Registered
Joined
·
5,094 Posts

Quote:
Originally Posted by krabapple /forum/post/12254364


I would say that no competently-built amp requires an outboard RF filter, but it's a fact that some 'high end' designs DO, and in those bizarre cases a cable with an RF filter has an advantage over one that doesn't...an 'unnatural' advantage, I'd say.

That never stopped a company like this to put one in anyhow
How else but a new gimmick to convince the gullible ones?
 

·
Registered
Joined
·
1,193 Posts
i want to thank my friends Joe and Ted for all their help yesterday; they both had sore fingers, knees and backs after all the cable switching. i particularly want to thank Chris; first of all for taking the time to help, for contributing his ideas to come up with the protocol, for being open-minded, but especially for his detailed, complete and accurate description of what happened. i type real slow and would have really struggeled to answer all the 'what happened?' questions.....Chris's write up and followup posts have saved me a good deal of trouble.


sorry for the delay in posting my perceptions.....i must admit to at least a bit of reluctance in discussing what happened......as the result was clearly much different than i expected. i do not feel bad about doing the test or the results (other than a slightly bruised ego) since i do feel it is the truth......at least the truth as to the protocol we used.


yes; i have, to some degree, changed my perspective on cable differences....but...my mind is still processing the results and what they mean for me. i hope that i can coherently relate the various thoughts that go thru my mind.


as Chris mentioned; the controls were successful at keeping me from knowing which cable was which. for each test i felt confident about my choice (except #6...see below).


my methodology was to use a demo disc with multiple cuts and switch quickly from one cut to another; listening to a cut from a few seconds to maybe 30 seconds.....coming back to the same cut a few times. i mostly used 5 or 6 cuts which (i felt) told me about various sonic issues. until the test was over i felt confident that i could hear real differences and that i had a 'lock' on which characteristic was which cable.


the tests were as follows;


1-Opus first.....i chose Opus first.

2-Monster first....i chose Opus first

3-Monster first....i chose Opus first.

4-Opus first....i chose Monster first

5-Opus first....i chose Opus first


after the first 3 tests i was getting a bit tired and the ear protection started causing a slight headache. at no time did i feel that was affecting my listening. after 5 tests i requested a break, and we went upstairs to play a bit of pool.


as we started the 6th test i could tell i was struggeling to hear differences. so i asked that i change CD's and re-do a 'sighted' test. i did not take my covered glasses off but they did tell me which cable was playing.


6-thrown out


then we did two more tests.


7-Opus first....i chose Opus first

8-Opus first....i chose Monster first.


when i made my choice known for #8 i was confident that i was 100% for all 7. then my friend Ted said 'that's it.....test over'. we had discussed prior that any result 7 out of 10 or better or 15 out of 20 or better would mean a positive result and to continue. once we got to only 3 out of 7 it was clear that we were not going to get a positive result.


OK......so how do i feel different?


about 1000 posts ago in the Randi thread Michael Grant asked me about how i would feel if i failed a blind test with my Opus cables.......would my opinion change?


my recollection is that i said that it would cause me to further investigate the issues with a much more skeptical viewpoint......but that until i did that additional investigation i would not dramatically alter my viewpoint. i think that comes closest to describing my feelings right now. i have some work to do. there is a chink in my subjective armor but it's not completely broken.


why did i fail?.....or put another way.....why did this test show no real difference?


was i overconfident?


yes; regardless of the eventual answer i was not respectful enough of the challenge.


looking at the specific test results i got both #1 and #7 correct......directly after the sighted trial where my aural memeory was the freshist. what if i were to have 'A' , then 'B', then 'X' every time? would that change the outcome?


did i spend enough time listening to the Monster prior to the test? i was very confident going in but now i think i had not taken enough time to really get into the Monster more.


i did not practice enough blinded.....for sure. if i had practiced i would know that my methodolgy would not show any difference. it's also possible that no amount of practice would be able to prove i could hear the differences blinded.


in my mind i am not confident that i will ever be able to hear reliable differences between the Monster and the Opus to pass a Blind test. OTOH i am also not sure i won't be able to do it.


and that is where i'm at right now.


i have lots more thoughts.....but as i said i type slow and i wanted to get something out.....i will do my best to answer any questions and add other thoughts.


and thanks everyone for all the positive comments and keeping to the high road.
 

·
Premium Member
Joined
·
2,247 Posts

Quote:
Originally Posted by CharlesJ /forum/post/12254894


Yes, but that doesn't stop hearing differences under sighted conditions with 1 minute or more of setup time


But you are correct, acoustic memory has been shown to be very short, on the order of seconds. So, even long segments of music is detrimental.


It should be interesting to hear from MikeL regarding how he perceived the sound quality from his system during the days following the test. Will it sound as good as percieved from the sighted Opus or worse as noted with the Monster cable. Was alleged placebo adding to or subtracting from the experience.
 

·
Registered
Joined
·
1,193 Posts

Quote:
Originally Posted by Hughman /forum/post/12255064


It should be interesting to hear from MikeL regarding how he perceived the sound quality from his system during the days following the test. Will it sound as good as percieved from the sighted Opus or worse as noted with the Monster cable. Was alleged placebo adding to or subtracting from the experience.

the system was sounding great last night after the test and this morning again.....in fact, as good as or better than ever.


i have not re-inserted the Monster since the test.


i need to just enjoy for awhile before i climb back on the 'cable-wars horse'. this is all a fun thing for me......not work. i need to re-charge my investigative juices.
 
21 - 40 of 384 Posts
Top