Validty of blind testing - Page 10 - AVS Forum
Forum Jump: 
Closed Thread
 
Thread Tools
post #271 of 355 Old 02-06-2014, 08:08 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by mcnarus View Post

Who's "relying" on it, and for what?

All we're saying is that you have biases you are not aware of, and that these biases must be controlled for if a listening test is to have any validity whatsoever.

My point is that this claim is made without understanding how bias via the subconscious works in the first place. How can you make the claim that a proper listening test depends on controlling or eliminating something which by every scientific account is thoroughly not understood?

Can you say this bias is the same for every person?
Can you say the degree of awareness vs unawareness of these biases is the same for every person?
Can you say some people are not aware of their biases?
Can you say subconscious biases have the same perceptual effect or influence on all people?
Can you say this bias even has an effect on some people?

No, of course not.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
Sponsored Links
Advertisement
 
post #272 of 355 Old 02-06-2014, 08:22 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,235
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 873 Post(s)
Liked: 518
Quote:
Originally Posted by arnyk View Post

Carlstrom, David, Greenhill, Laurence, Krueger, Arnold, "Some Amplifiers Do Sound Different", The Audio Amateur, 3/82, p. 30, 31, also reprinted in Hi-Fi News & Record Review, Link House Magazines, United Kingdom, Dec 1982, p. 37.
Thank you. So to state the obvious, in 31 years since, after thousands upon thousands of posts in many audio forums, you have never explained how you personally, without anyone else involved, created a double blind test and ran it? Have you not run any double blind tests since then that you documented some online forum we can look at?

Didn't you say you have run more blind tests than anyone but JJ here? How did you avoid talking about their specifics in three decades?

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #273 of 355 Old 02-06-2014, 08:25 AM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 3,988
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 107 Post(s)
Liked: 157
If the bias comes from knowing, then not knowing removes the source of the bias and how it works with any particular individual is rendered moot. SImple logic, it seems.

While this bias (a person could google "placebo effect") has been understood to exist for a few decades (yes, it's old old news) IDK if anybody has figured out exactly how it works.

I don't think it is necessary to understand how it works. Moreover, the batch of stuff that psychologists etc. call "subconscious" is, AFAIK, not well understood in general. It's clearly there, but since no human has direct access to what's going on in their subconscious (onaccounta it's subconscoious - - not within the conscious range) it's a little hard to tease out what's really going on.

But just like Newtonian mechanics works just fine at the sizes we encounter in everyday life, although it's premises are not regarded as accurate anymore (e.g. the pool balls never actually touch each other, so the force is transferred through a different mechanism) we still rely on Newtonian precepts every day, because they work just fine for regular folks' needs.

See? You can understand a phenomenon to the point of being able to mathematically calculate pretty much everything we care to, even while misunderstanding underlying principals.

And I don't need string theory or any other esoteric science to tell me that if I can't turn the nut, I should try a longer handle. Pure Newton. Works great.
JHAz is online now  
post #274 of 355 Old 02-06-2014, 08:33 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by JHAz View Post

If the bias comes from knowing, then not knowing removes the source of the bias and how it works with any particular individual is rendered moot. SImple logic, it seems.

While this bias (a person could google "placebo effect") has been understood to exist for a few decades (yes, it's old old news) IDK if anybody has figured out exactly how it works.

I don't think it is necessary to understand how it works. Moreover, the batch of stuff that psychologists etc. call "subconscious" is, AFAIK, not well understood in general. It's clearly there, but since no human has direct access to what's going on in their subconscious (onaccounta it's subconscoious - - not within the conscious range) it's a little hard to tease out what's really going on.

But just like Newtonian mechanics works just fine at the sizes we encounter in everyday life, although it's premises are not regarded as accurate anymore (e.g. the pool balls never actually touch each other, so the force is transferred through a different mechanism) we still rely on Newtonian precepts every day, because they work just fine for regular folks' needs.

See? You can understand a phenomenon to the point of being able to mathematically calculate pretty much everything we care to, even while misunderstanding underlying principals.

And I don't need string theory or any other esoteric science to tell me that if I can't turn the nut, I should try a longer handle. Pure Newton. Works great.

Are you really claiming our understanding of the subconscious is equivalent to Newton's law of gravity? Really?

The problem here is that many of you are trying to apply something that MAYBE true generally, and let me emphasize MAYBE, and claim its true on each and every person on a specific basis and that it works in exactly the same way for each.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #275 of 355 Old 02-06-2014, 08:34 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,074
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 368 Post(s)
Liked: 764
Quote:
Originally Posted by jim19611961 View Post

My point is that this claim is made without understanding how bias via the subconscious works in the first place. How can you make the claim that a proper listening test depends on controlling or eliminating something which by every scientific account is thoroughly not understood?

Can you say this bias is the same for every person?
Can you say the degree of awareness vs unawareness of these biases is the same for every person?
Can you say some people are not aware of their biases?
Can you say subconscious biases have the same perceptual effect or influence on all people?
Can you say this bias even has an effect on some people?

No, of course not.

The reason we know about hearing bias is that some audible differences disappear when we control bias. It is reasonable to say that science doesn't understand biases completely. It is unreasonable to suggest that, when audible differences disappear, the disappearance isn't representative of a valid result. In fact it is illogical to suggest that.

As to your litany of questions, i can tell you that the answer to #3 is yes, In fact I would say that most people are not aware of their hearing bias. I agree on the others. As an example, after a couple of years of bias controlled testing my biases have changed. If I listen to a sighted comparison of two DACs, I wouldn't hear a subtle difference between them if one actually existed simply because my expectation is that there would be no difference. A bias controlled test would sort that out in the same way it would sort out the opposite bias.

But these differences in bias aren't germaine to the issue. With DBT's we control the bias, regardless of how much there might be. How do we know? We tend to get the same results from everyone that we test.
FMW is offline  
post #276 of 355 Old 02-06-2014, 08:40 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,235
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 873 Post(s)
Liked: 518
Quote:
Originally Posted by BruceD View Post

I just don't see how any point of view championed by an entity that always has a vested MONETARY interest in pushing high end audio can be taken seriously.

I'm in the camp of not sharing a tent and NOT believing anything from amirm, because his business is selling high end audio.

Fundamentally, I think he is part of the problem with respect to what is wrong with audio these days.
No, lack of research and understanding the facts and science is the problem with audio in forums. It has always been there and continues here. Folks don't have a technical answer to provide so try to change the topic to poster. Here is an answer to your accusation so hopefully it is enough and we can discuss OP's topic again rather than me.

To me saying, "I have been a member here for far longer than I have had my business. My posts in these areas have been consistent for years. Feel free to go and ask. Chu actually knows and I am sure would chime in if I had said something inconsistent."

Chu kindly answered with this:
Quote:
Originally Posted by Chu Gai View Post

Amir speaks the truth. Long before he entered into an audio business related venture, his comments on matters such as those in this thread have been consistent. Myself, I don't see his participation in these sorts of discussions as sales shacking. If part of what high-end is supposed to mean is the most faithful, do as little damage as possible to the signal regardless of whether it's audible or not, I can appreciate his position. I can also appreciate that with judicious shopping, those of us who are financially constrained in this hobby can put put together a system that was unimaginable 30-40 years ago. I hope Amir continues to post and continues to find the time to respond to questions as well as offer his perspective.

Yes, I am a founder of my company, Madrona Digital. I created it because the local AV shops focused on what has margin, rather than what has performance. I don't make a living from my company at all even though it does and has created jobs for our employees. With one exception, no one from this forum ever come to me to buy anything. And that one instance was a friend. Even in that instance I insisted that he buy from a local dealer but he said there were none.

Now, maybe you can link to web sites of any company that sells high-end that talks about how double blind tests are essential part of audio research as I have. And then you can look at examples on my company's site which I personally manage: http://www.madronadigital.com/Showroom/HomeTheater.html

"You may have heard of room optimization logic that exists in traditional mass market products. Alas, while they can improve the audio at times, their capability is highly limited due to cost restrictions. Blind testing done by Harman, the parent company of JBL, shows that some of these systems are actually worse than doing nothing at all!"

Link me to posts from the company founders that insist on value of double blind testing as I have here across hundreds of posts.

Of course folks forget that AVS Forum is owned by one of the largest AV firms in US. Why not get suspicious about that if you want to imagine problems with audio industry rather than answering simple technical questions?

Here is hoping in your next post you document a double blind test that you have created and ran so that we can discuss the merits of it.

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #277 of 355 Old 02-06-2014, 08:43 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by FMW View Post

The reason we know about hearing bias is that some audible differences disappear when we control bias. It is reasonable to say that science doesn't understand biases completely. It is unreasonable to suggest that, when audible differences disappear, the disappearance isn't representative of a valid result. In fact it is illogical to suggest that.

As to your litany of questions, i can tell you that the answer to #3 is yes, In fact I would say that most people are not aware of their hearing bias. I agree on the others. As an example, after a couple of years of bias controlled testing my biases have changed. If I listen to a sighted comparison of two DACs, I wouldn't hear a subtle difference between them if one actually existed simply because my expectation is that there would be no difference. A bias controlled test would sort that out in the same way it would sort out the opposite bias.

But these differences in bias aren't germaine to the issue. With DBT's we control the bias, regardless of how much there might be. How do we know? We tend to get the same results from everyone that we test.

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one? One that would be noticeable in a sighted test?

I am perfectly fine to say I dont know this answer. I think its pretty clear no one could.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #278 of 355 Old 02-06-2014, 08:58 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,074
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 368 Post(s)
Liked: 764
Quote:
Originally Posted by jim19611961 View Post

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one? One that would be noticeable in a sighted test?

I am perfectly fine to say I dont know this answer. I think its pretty clear no one could.

I think you might be better off just dealing with the test results rather than trying to skew science to try to disprove them or to fit your preferences. Give me a logical reason that controlling bias (however bias works) doesn't do a better job of measuring audibility than a sighted test. That is where you run into trouble. There is no point in telling us that scientific knowledge is incomplete. We all know that and agree that it is true. You need to deal with what we do know, not what we don't know. What we do know is that some audible differences disappear when we control bias. We actually know that beyond any doubt. Deal with what we know. It is waste of time dealing with what we don't know.
FMW is offline  
post #279 of 355 Old 02-06-2014, 09:02 AM
AVS Special Member
 
JHAz's Avatar
 
Join Date: Mar 2009
Posts: 3,988
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 107 Post(s)
Liked: 157
Quote:
Originally Posted by jim19611961 View Post

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one? One that would be noticeable in a sighted test?

I am perfectly fine to say I dont know this answer. I think its pretty clear no one could.

google placebo effect. google double blind testing. It's old science by now. Analytically, if we KNOW that knowledge of what one is testing skews the results, don't we have to know they're skewed versus the condition of not knowing? You have to start somewhere . . .

Your suggestion would conclude that blind testing of pharmaceuticals is also hopeless, so everybody should give up on medicines that are sold through the pharmacy (the hell with chemo) and buy stuff from the back of the wagon of the guy whose spiel you like the best. I'm guessing recovery rates and survival rates would plummet.

Purely anecdotally, i find that if I go to a concert where I like the sound but don't recognize the speakers, can't find the amp racks, and can't get close enough to see what board they're using FOH, I still can hear The trick for me is to stop wondering (also "what amp is that guitarist playing through?") and just enjoy the performance.
JHAz is online now  
post #280 of 355 Old 02-06-2014, 09:04 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by FMW View Post

I think you might be better off just dealing with the test results rather than trying to skew science to try to disprove them or to fit your preferences. Give me a logical reason that controlling bias (however bias works) doesn't do a better job of measuring audibility than a sighted test. That is where you run into trouble. There is no point in telling us that scientific knowledge is incomplete. We all know that and agree that it is true. You need to deal with what we do know, not what we don't know. What we do know is that some audible differences disappear when we control bias. We actually know that beyond any doubt. Deal with what we know. It is waste of time dealing with what we don't know.

What we think we know rests on understanding on all the related matters. Often times in science, what we don't know about something, once it becomes known, sheds new light on what we thought we knew previously. And sometimes, it makes us completely rethink what we thought we knew.

The fact that some audible differences disappear sometimes under a certain control condition suggests a incomplete picture to me. I am not sure if DBT does a better job vs sighted tests or not. The jury is still out for me. It does provide constraints in the test that in some ways are beneficial, I agree. But we dont know if removing the sighting doesn't create other effects that may be detrimental and are so far either not understood or unobserved.

I am not sure I have a preference. So I am not trying to fit anything.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #281 of 355 Old 02-06-2014, 09:08 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by JHAz View Post

google placebo effect. google double blind testing. It's old science by now. Analytically, if we KNOW that knowledge of what one is testing skews the results, don't we have to know they're skewed versus the condition of not knowing? You have to start somewhere . . .

Your suggestion would conclude that blind testing of pharmaceuticals is also hopeless, so everybody should give up on medicines that are sold through the pharmacy (the hell with chemo) and buy stuff from the back of the wagon of the guy whose spiel you like the best. I'm guessing recovery rates and survival rates would plummet.

Purely anecdotally, i find that if I go to a concert where I like the sound but don't recognize the speakers, can't find the amp racks, and can't get close enough to see what board they're using FOH, I still can hear The trick for me is to stop wondering (also "what amp is that guitarist playing through?") and just enjoy the performance.



Please stop trying to compare audio listening test procedures to pharmaceutical drugs. Apples and oranges.
amirm likes this.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #282 of 355 Old 02-06-2014, 09:14 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,235
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 873 Post(s)
Liked: 518
Quote:
Originally Posted by FMW View Post

The reason we know about hearing bias is that some audible differences disappear when we control bias.
True.
Quote:
It is reasonable to say that science doesn't understand biases completely. It is unreasonable to suggest that, when audible differences disappear, the disappearance isn't representative of a valid result. In fact it is illogical to suggest that.
Only if your blind and sighted tests are identical in all aspects. Change anything and then it is your responsibility is to prove it did not change the results. The results are only valid if we can do that. Blindly believing the outcome will most definitely create the potential for invalid results. See earlier point for example for lack of control.
Quote:
But these differences in bias aren't germaine to the issue. With DBT's we control the bias, regardless of how much there might be. How do we know? We tend to get the same results from everyone that we test.
We only know that for large differences. For small differences that are non-linear, that is not true at all. Here is yet another personal example smile.gif :

The time is around year 2000 and the music industry wants to release content for DVD audio and likes to embed a watermark in them. A watermark as you can imagine is supposed to be "invisible." In the case of audio, it means inaudible. Since the application of DVD-A was high resolution audio, maintaining fidelity was paramount. So an RFP was put out for the industry to propose their watermark algorithm to be evaluated. I was working at Microsoft and had a close relationship with Microsoft Research (MSR). They had been working on a watermark algorithm and wanted us to propose it. We said fine. We get near the deadline and I have not seen or heard the technology. I ask if they are ready to submit and they say yes. I ask about the quality and they say they have tested with a number of people already and none could detect the inserted bits. I asked if I could listen and do a sanity check and they said sure. They provided two files one before and the other post watermark insertion. They did not tell me which file was which. The track was 24-bits, 96 Khz and was about 3 to 4 minutes.

Just a bit of background. A watermark is a handful of bits, usually 32 bits or so in such application. With error correction and anti-tamper it is probably a few thousand bits spread across the whole song. The job of a good watermark algorithm then is to change the audio samples in a way that they still carry the message but do not change the fidelity. The general technique is called "data hiding." The sample file I was given was about 500 million bits per channel. So one would think find a few thousand bits that you could change in that massive sea of bits would be easy. The technology uses a psychoacoustic model of the ear and searches for areas of music where the change would easily be masked by the music.

Back to the test, I played the songs and boy, it seemed impossible to find the impact of the watermark. Where would I look? Which of the two files would I compare? So I started to listen to the songs from start to finish. Did this a number of times, focusing on transients only. This filtered out 99% of the music. I figured it would be very easy for them to hide bits in the other segments in an inaudible way. I grabbed one of the transients and created a tiny, sub-second loop that I could compare instantly to the other file in the same manner. And there it was. A very subtle change.

I go back to our researcher and tell them which file is degraded down to the exact millisecond accuracy of where that transient was. He just about fell off his chair! I thought he would be surprised but not that surprised! Job was not done however. They went back and looked at the algorithm and realized where it had erred in its psychoacoustic model and made a change. I did another listen and the problem was gone.

This is not the type of problem that would have been remotely found the general public. I could get 1000 people and they would all vote them the same. What enabled me to find the problem was the knowledge of how the system worked and where its weakness might be. Others do not have this knowledge so face a far, far taller mountain.

So no, when it comes to smaller differences that are non-linear, we don't all end in the same place.

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #283 of 355 Old 02-06-2014, 09:34 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by arnyk View Post

Carlstrom, David, Greenhill, Laurence, Krueger, Arnold, "Some Amplifiers Do Sound Different", The Audio Amateur, 3/82, p. 30, 31, also reprinted in Hi-Fi News & Record Review, Link House Magazines, United Kingdom, Dec 1982, p. 37.
Thank you. So to state the obvious, in 31 years since, after thousands upon thousands of posts in many audio forums, you have never explained how you personally, without anyone else involved, created a double blind test and ran it? Have you not run any double blind tests since then that you documented some online forum we can look at?

Didn't you say you have run more blind tests than anyone but JJ here? How did you avoid talking about their specifics in three decades?

The simple answer is that unlike some people, I don't write a magazine aritcle every time I blow my nose. ;-)

Amir, I provided you with a undeniable answer for your question.

Before I accede to any more of your threats and demands underscored as they are with undertones of incredulity and apparent attacks on my credibility, please honor my humble offering by providing a piece of evidence of equal or superior stature that shows that you have ever personally organized and participated even just one DBT and that similarly specifies the procedures you used.
arnyk is offline  
post #284 of 355 Old 02-06-2014, 09:36 AM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,167
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 410
Quote:
Originally Posted by BruceD View Post

I just don't see how any point of view championed by an entity that always has a vested MONETARY interest in pushing high end audio can be taken seriously.

I'm in the camp of not sharing a tent and NOT believing anything from amirm, because his business is selling high end audio.

Fundamentally, I think he is part of the problem with respect to what is wrong with audio these days.
Careful, Bruce. amir will accuse you of being "too emotional", thus attempting to divert attention from the topic at hand.
RobertR is offline  
post #285 of 355 Old 02-06-2014, 09:40 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,074
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 368 Post(s)
Liked: 764
Quote:
Originally Posted by arnyk View Post

The simple answer is that unlike some people, I don't write a magazine aritcle every time I blow my nose. ;-)

Amir, I provided you with a undeniable answer for your question.

Before I accede to any more of your threats and demands underscored as they are with undertones of incredulity and apparent attacks on my credibility, please honor my humble offering by providing a piece of evidence of equal or superior stature that shows that you have ever personally organized and participated even just one DBT and that similarly specifies the procedures you used.

Arny you are dealing with someone who enjoys argument for the sake of argument. He will argue against anything regardless of his own position. Nothing is going to change that. My advice is to put him on ignore and spend your forum time on more useful pursuits.
FMW is offline  
post #286 of 355 Old 02-06-2014, 09:44 AM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,167
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 410
Quote:
Originally Posted by jim19611961 View Post

Since its clear we don't understand how subconscious bias works on an individual basis, how do we know that blinding people doesn't cause a specific suggestion in the subconscious that there is no difference when in fact there is one? One that would be noticeable in a sighted test?
Experience tells us this is not the case. Dedicated subjectivists don't change their biases when participating in a DBT. They're motivated to prove they're right about the invalidity of DBTs.
RobertR is offline  
post #287 of 355 Old 02-06-2014, 09:45 AM
Super Moderator
 
markrubin's Avatar
 
Join Date: May 2001
Location: Jersey Shore
Posts: 23,014
Mentioned: 1 Post(s)
Tagged: 0 Thread(s)
Quoted: 114 Post(s)
Liked: 419
Moderator

please do not insult another member
markrubin is offline  
post #288 of 355 Old 02-06-2014, 09:46 AM
AVS Special Member
 
RobertR's Avatar
 
Join Date: Dec 2000
Location: California
Posts: 6,167
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 182 Post(s)
Liked: 410
Quote:
Originally Posted by FMW View Post

Arny you are dealing with someone who enjoys argument for the sake of argument. He will argue against anything regardless of his own position.
Yes, he'll even claim he is on your side while simultaneously saying you're wrong.
RobertR is offline  
post #289 of 355 Old 02-06-2014, 09:46 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by amirm View Post

Well said. Heaven knows we have seen our share of emotional outbursts in this thread. The only problem is: the emotional side is only being shown by folks in your camp. Can you explain that?

There are many different emotions. One of them is fear. It seems to me that only fear can explain why a certain person has been failing to answer reasonable questions about his evaluations of jitter by means of sighted evaluations and cable swapping, which he frequently faults others for when they do it.

Just to jog your memory:

http://www.avsforum.com/t/1125344/jim-burns-are-some-blu-ray-players-better-on-hdmi-hd-picture-quality/150#post_15990901
Quote:
Originally Posted by AmirM 
"Many people, myself included, can hear something that is only explained by jitter. When I change S/PDIF cables, I hear the difference even with high-end DACs. Hard to imagine anything else being responsible for the difference."
arnyk is offline  
post #290 of 355 Old 02-06-2014, 09:51 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,074
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 368 Post(s)
Liked: 764
Quote:
Originally Posted by jim19611961 View Post

What we think we know rests on understanding on all the related matters.

I see. You believe that no knowledge is valid unless we have knowledge of everything. I wouldn't spend a lot of time trying to convince a scientist of that.
Quote:
Often times in science, what we don't know about something, once it becomes known, sheds new light on what we thought we knew previously. And sometimes, it makes us completely rethink what we thought we knew.

Yes, that's true. So you want to wait until the end of time before you accept any knowledge as valid?
Quote:
The fact that some audible differences disappear sometimes under a certain control condition suggests a incomplete picture to me.

What picture are you trying to paint? The disappearance of audible differences occur every time with every listener. Putting qualifiers on something that doesn't warrant qualification isn't a very strong argument.
Quote:
I am not sure if DBT does a better job vs sighted tests or not. The jury is still out for me. It does provide constraints in the test that in some ways are beneficial, I agree. But we dont know if removing the sighting doesn't create other effects that may be detrimental and are so far either not understood or unobserved.

I am not sure I have a preference. So I am not trying to fit anything.

Your preference is painted all over everything you write. You don't like bias controlled listening tests. I get it. There are a lot of people with the same sentiment. But it is still an illogical position. It is belief vs. fact. Preference vs. fact. You can take any side. But fighting facts with belief is an incredibly weak argument.
FMW is offline  
post #291 of 355 Old 02-06-2014, 10:00 AM
AVS Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,300
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 127 Post(s)
Liked: 183
Quote:
Originally Posted by RobertR View Post

Those kind of declarative statements (the kind you said you've made) don't include qualifiers about "unreliable memory." It's only when subject to the discipline of this discussion that you seem compelled to include them.

The lack of qualifiers -- which you almost always see in scientific literature, e.g., 'our results make it likely that / our results suggest " that -- is one of the marks of a classic 'audiophile' claim. You rarely see acknowledgement that they *might* have heard wrong, *might* have been influenced by non-audio factors....
things which in science, are a *given*. To be acknowledged and dealt with so that their impact on data analysis is minimized.
CruelInventions likes this.
krabapple is offline  
post #292 of 355 Old 02-06-2014, 10:00 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by jim19611961 View Post

Quote:
Originally Posted by FMW View Post

I think you might be better off just dealing with the test results rather than trying to skew science to try to disprove them or to fit your preferences. Give me a logical reason that controlling bias (however bias works) doesn't do a better job of measuring audibility than a sighted test. That is where you run into trouble. There is no point in telling us that scientific knowledge is incomplete. We all know that and agree that it is true. You need to deal with what we do know, not what we don't know. What we do know is that some audible differences disappear when we control bias. We actually know that beyond any doubt. Deal with what we know. It is waste of time dealing with what we don't know.

What we think we know rests on understanding on all the related matters. Often times in science, what we don't know about something, once it becomes known, sheds new light on what we thought we knew previously. And sometimes, it makes us completely rethink what we thought we knew.

The fact that some audible differences disappear sometimes under a certain control condition suggests a incomplete picture to me. I am not sure if DBT does a better job vs sighted tests or not. The jury is still out for me. It does provide constraints in the test that in some ways are beneficial, I agree. But we dont know if removing the sighting doesn't create other effects that may be detrimental and are so far either not understood or unobserved.

I am not sure I have a preference. So I am not trying to fit anything.

It would seem from the above that some are under the impression that audio DBTs were some new untried new thing that has never been generally accepted by various professional organizations as standard practice.

I'm wondering how one justifies proffering this viewpoint when audio DBTs have been generally accepted practice in the audio industry and among audio professionals for over 40 years?

Perhaps people believe that sell-appointed experts writing for consumer magazines and web sites somehow trump organizations such as the AES and EBU?
arnyk is offline  
post #293 of 355 Old 02-06-2014, 10:06 AM
AVS Special Member
 
jim19611961's Avatar
 
Join Date: Mar 2011
Location: Texas
Posts: 1,397
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 126 Post(s)
Liked: 85
Quote:
Originally Posted by FMW View Post

1) I see. You believe that no knowledge is valid unless we have knowledge of everything. I wouldn't spend a lot of time trying to convince a scientist of that.
2) Yes, that's true. So you want to wait until the end of time before you accept any knowledge as valid?
3) The disappearance of audible differences occur every time with every listener. Putting qualifiers on something that doesn't warrant qualification isn't a very strong argument.
4) Your preference is painted all over everything you write. You don't like bias controlled listening tests. I get it.
5) There are a lot of people with the same sentiment. But it is still an illogical position. It is belief vs. fact. Preference vs. fact. You can take any side. But fighting facts with belief is an incredibly weak argument.

1) No. The scientific method is sufficient. And that means the exact same conditions for each test with exactly the same outcome. NOT.
2) No. I am fine walking on a cement sidewalk without knowing the quantum state of each of its molecules biggrin.gif
3) Depends on the degree of the difference and what kind. Changing the temperature or humidity of the room makes some difference.
4) No. You dont get it.
5) Many believe Christ was resurrected. Is that a fact or belief?

Look let me sum it up this way. One of these statements I am onboard with. The other I am not.

A) People cannot discern audible differences using sighted tests. They are always wrong. DBT's are always right.

B) DBT's maybe more accurate than sighted tests.

My Room
My Music
Rega - Apollo, Rega - DAC, Goldpoint Passive, (2) Classe CA-100 bridged power amps (350w)
Jenzen Next
jim19611961 is offline  
post #294 of 355 Old 02-06-2014, 10:06 AM
AVS Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,300
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 127 Post(s)
Liked: 183
Quote:
Originally Posted by jim19611961 View Post

I thought one of the main arguments against sighted testing was being consciously aware of which component it was you were hearing.

You thought wrong.

Many an audiophile claims they are not significantly influenced by what they see, or know in advance. They consciously think they are free of such biases. They believe they are relying entirely on what their ears (or nose, or tongue, if we're talking wine tasters) tell them.

Years of sensory and product -choice testing data tell another tale on them.

It is not just consciously acknowledged or denied biases at work.

Btw, people typically claim to be nonracist and nonsexist, too. Very sincerely.

Yet when psychological testing is applied....we learn again and again that we aren't always consciously aware of our biases.
CruelInventions likes this.
krabapple is offline  
post #295 of 355 Old 02-06-2014, 10:10 AM
FMW
AVS Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 5,074
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 368 Post(s)
Liked: 764
Quote:
Originally Posted by arnyk View Post




Perhaps people believe that sell-appointed experts writing for consumer magazines and web sites somehow trump organizations such as the AES and EBU?

I don't think the high end audio industry would exist without the subjective magazine writers. Perhaps it never would have existed.
FMW is offline  
post #296 of 355 Old 02-06-2014, 10:11 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by jim19611961 View Post


Are you really claiming our understanding of the subconscious is equivalent to Newton's law of gravity? Really?

The problem here is that many of you are trying to apply something that MAYBE true generally, and let me emphasize MAYBE, and claim its true on each and every person on a specific basis and that it works in exactly the same way for each.

IME once a person actually does an ABX test and thinks introspectively about the experience of making forced choices based on only the evidence of their own hearing, many of these issues become crystal clear.

By repeatedly criticizing audio DBTs it seems obvious that you are trying to talk people out of believing in the necessity for their use when they are currently quite convinced of it based on their personal experiences.It seems like you have even tried this with the person who invented the ABX test over 40 years ago! It seems that you yourself have no experience with doing audio DBTs for the reason stated above. This itself seems pretty strange. Why are you doing this?
arnyk is offline  
post #297 of 355 Old 02-06-2014, 10:12 AM
AVS Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,300
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 127 Post(s)
Liked: 183
Quote:
Originally Posted by jim19611961 View Post

My point is that this claim is made without understanding how bias via the subconscious works in the first place. How can you make the claim that a proper listening test depends on controlling or eliminating something which by every scientific account is thoroughly not understood?

Good god.

Do you realize that science would come to a halt if this was a serious objection?

As a general rule, science employs *controls* to mitigate the effect of investigator bias. In studies involving humans, a further layer may be employed to mitigate the effects of bias in the subject. In a DBT, both layers are in place.

We are far from understanding how minds work. What level of understanding do *you* think we need, in order for psychological bias controls to be 'safe to use'?
CruelInventions likes this.
krabapple is offline  
post #298 of 355 Old 02-06-2014, 10:16 AM
AVS Addicted Member
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,235
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 873 Post(s)
Liked: 518
Quote:
Originally Posted by arnyk View Post

The simple answer is that unlike some people, I don't write a magazine aritcle every time I blow my nose. ;-)
But you post on forums before and after you blow your nose. biggrin.gif Surely a prolific poster like you that has the maniacal focus on double blind tests would have shared tests that he created and ran. I know in the 3 or so years I have debated you that you have not described any.
Quote:
Amir, I provided you with a undeniable answer for your question.
And it was an embarrassing answer. It was the work of three people and not just you. Looking for the back up that you have run more double blind tests than anyone but JJ.
Quote:
Before I accede to any more of your threats and demands underscored as they are with undertones of incredulity and apparent attacks on my credibility, please honor my humble offering by providing a piece of evidence of equal or superior stature that shows that you have ever personally organized and participated even just one DBT and that similarly specifies the procedures you used.
Not attacking your credibility. Want to examine your test protocols to see if they are consistent with how the science says we should administer them. As for me, it is a strange question because I have documented a number of them. The first one was in post #16: http://www.avsforum.com/t/1515576/validty-of-blind-testing#post_24302549. I explained the same in our first debate thread where I quoted Chu.

Here is another one: when I was at Microsoft we acquired Pacific Microsonics (PMI). Not for the HDCD technology but for their speaker correction system (wanted to make cheap computer speakers sound better). Being an audiophile, I wanted to determine how good HDCD was. So I asked for their reference tracks before and after conversion to HDCD (i.e. 16 bit vs HDCD encoded 20 bits). I was pleasantly surprised that I could easily tell the extra "air" and resolution. One of the PMI testers sat a few offices down from me. So I go there and i say, "hey, I just listened to HDCD and boy, it sounds good." To my surprised he said, "you don't really buy into that pseudo science, do you?" I was like, what? You don't think it works? I explained that I had just tested a bunch and the improvement was real. He said, "OK, let's test you again." He gives me his headphones and asks me to turn around. He plays one sample and then the other. I heard exactly the improvement I had heard before. It didn't even require concentrating! I turn confidently to him and say which one was the HDCD. The son of a you know what tells me, "I was playing the same 16 bit CD track over and over again!" So here I was telling him two identical files sounded different. You want to know why I believe in double blind testing? This is one fo the reasons.

As you see Arny I am very transparent with my experiences whether they support certain point of view or not. To wit, I have documented the above on forums and then some. So I ask again, why is there such paucity of your double blind testing and protocols online?

Amir
Retired Technology Insider
Founder, Madrona Digital
"Insist on Quality Engineering"
amirm is offline  
post #299 of 355 Old 02-06-2014, 10:18 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by FMW View Post

Quote:
Originally Posted by arnyk View Post


Perhaps people believe that sell-appointed experts writing for consumer magazines and web sites somehow trump organizations such as the AES and EBU?

I don't think the high end audio industry would exist without the subjective magazine writers. Perhaps it never would have existed.

There is something to what you say.

I was there at the beginning, and I would like to see Harry Pearson and J. Gordon Holt face each other down and settle which one invented the high end audio business as we know it.

It is clear that JGH founded SP in 1962 and HP founded TAS in 1973, but the question in my mind is whether SP was really about high end audio as we know it. Seems too objectivist. It did include measurements.
arnyk is offline  
post #300 of 355 Old 02-06-2014, 10:20 AM
AVS Addicted Member
 
arnyk's Avatar
 
Join Date: Oct 2002
Location: Grosse Pointe Woods, MI
Posts: 14,382
Mentioned: 2 Post(s)
Tagged: 0 Thread(s)
Quoted: 753 Post(s)
Liked: 1167
Quote:
Originally Posted by amirm View Post

Quote:
Originally Posted by arnyk View Post

The simple answer is that unlike some people, I don't write a magazine aritcle every time I blow my nose. ;-)
But you post on forums before and after you blow your nose. biggrin.gif Surely a prolific poster like you that has the maniacal focus on double blind tests would have shared tests that he created and ran. I know in the 3 or so years I have debated you that you have not described any.
Quote:
Amir, I provided you with a undeniable answer for your question.
And it was an embarrassing answer. It was the work of three people and not just you. Looking for the back up that you have run more double blind tests than anyone but JJ.
Quote:
Before I accede to any more of your threats and demands underscored as they are with undertones of incredulity and apparent attacks on my credibility, please honor my humble offering by providing a piece of evidence of equal or superior stature that shows that you have ever personally organized and participated even just one DBT and that similarly specifies the procedures you used.
Not attacking your credibility. Want to examine your test protocols to see if they are consistent with how the science says we should administer them. As for me, it is a strange question because I have documented a number of them. The first one was in post #16: http://www.avsforum.com/t/1515576/validty-of-blind-testing#post_24302549. I explained the same in our first debate thread where I quoted Chu.

Here is another one: when I was at Microsoft we acquired Pacific Microsonics (PMI). Not for the HDCD technology but for their speaker correction system (wanted to make cheap computer speakers sound better). Being an audiophile, I wanted to determine how good HDCD was. So I asked for their reference tracks before and after conversion to HDCD (i.e. 16 bit vs HDCD encoded 20 bits). I was pleasantly surprised that I could easily tell the extra "air" and resolution. One of the PMI testers sat a few offices down from me. So I go there and i say, "hey, I just listened to HDCD and boy, it sounds good." To my surprised he said, "you don't really buy into that pseudo science, do you?" I was like, what? You don't think it works? I explained that I had just tested a bunch and the improvement was real. He said, "OK, let's test you again." He gives me his headphones and asks me to turn around. He plays one sample and then the other. I heard exactly the improvement I had heard before. It didn't even require concentrating! I turn confidently to him and say which one was the HDCD. The son of a you know what tells me, "I was playing the same 16 bit CD track over and over again!" So here I was telling him two identical files sounded different. You want to know why I believe in double blind testing? This is one fo the reasons.

As you see Arny I am very transparent with my experiences whether they support certain point of view or not. To wit, I have documented the above on forums and then some. So I ask again, why is there such paucity of your double blind testing and protocols online?

I see none of the evidence that I asked for, and I do see more insults, such as the claim that the magazine article that I cited was "embarassing". It is quite obvious that my procedures for DBTs were published in Clark's landmark article about ABX since it mentions me as a contributor. Ask him what I contributed. There is a second set of documentation of my prodcedures for doing DBTs in the PCABX web site as archived in the Wayback machine. Since you may lack the wherewithall to access it, I will post a major component of its documentation at the bottom of this replay.

I see no documentation of your claims. Therefore i have every reason to doubt the responsiveness and good faith of this reply. No deal.



Ten (10) Requirements For Sensitive and Reliable Listening Tests

(1) Program material must include critical passages that enable audible differences to be most easily heard.

(2) Listeners must be sensitized to a audible differences, so that if an audible difference is generated by the equipment, the listener will notice it and have a useful reaction to it.

(3) Listeners must be trained to listen systematically so that audible problems are heard.

(4) Procedures should be "open" to detecting problems that aren't necessarily technically well-understood or even expected, at this time. A classic problem with measurements and some listening tests is that each one focuses on one or only a few problems, allowing others to escape notice.

(5) We must have confidence that the Unit Under Test (UUT) is representative of the kind of equipment it represents. In other words the UUT must not be broken, it must not be appreciably modified in some secret way, and must not be the wrong make or model, among other things.

(6) A suitable listening environment must be provided. It can't be too dull, too bright, too noisy, too reverberant, or too harsh. The speakers and other components have to be sufficiently free from distortion, the room must be noise-free, etc..

(7) Listeners need to be in a good mood for listening, in good physical condition (no blocked-up ears!), and be well-trained for hearing deficiencies in the reproduced sound.

(8) Sample volume levels need to be matched to each other or else the listeners will perceive differences that are simply due to volume differences.

(9) Non-audible influences need to be controlled so that the listener reaches his conclusions due to "Just listening".

(10) Listeners should control as many of the aspects of the listening test as possible. Self-controlled tests usually facilitate this. Most importantly, they should be able to switch among the alternatives at times of their choosing. The switchover should be as instantaneous and non-disruptive as possible.
arnyk is offline  
Closed Thread Audio theory, Setup and Chat

User Tag List

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off