The Dishonesty of Sighted Listening Tests - by Sean Olive - Page 8 - AVS Forum | Home Theater Discussions And Reviews
Forum Jump: 
 159Likes
 
Thread Tools
post #211 of 525 Old 01-26-2015, 01:01 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
And, even if expert, trained listeners submit to multiple tests, like Amir, you claim lying & cheating - really?
Well, one never knows, does one?
krabapple is offline  
Sponsored Links
Advertisement
 
post #212 of 525 Old 01-26-2015, 01:09 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
Quote:
Originally Posted by krabapple View Post
Certainly there are cases where A and B really do not display an aujdible difference. (You can even devise a test where they *cannot possibly* be different -- a 'phantom switch' paradigm). But a subject doesn't *know* there is no difference. So the subject still scans for difference. That's work too -- arguably the hardest work for a DBT. Have you never done a DBT?
It's called a hidden control & if you knew anything about blind testing you would be aware of it. from ITU document:
Quote:
A major consideration is the inclusion of appropriate control conditions. Typically, control conditions include the presentation of unimpaired audio materials, introduced in ways that are unpredictable to the subjects. It is the differences between judgement of these control stimuli and the potentially impaired ones that allows one to conclude that the grades are actual assessments of the impairments.
Quote:
If you think no one has ever reports 'it took me a few trials to 'home in' on the difference, but once I did that, the rest of the test was easy', you should talk to Amir.

Hope it doesn't blow your gaskets too much but there's this too: often, people (given the chance) will try to find a segment of the probe signal that most readily displays a difference - the 'tell'. That takes 'work' too. Ask Amir about that too if you disbelieve me. He knows everything.
Yes, all that is pre-test training - not done during the test trials, please. Every statement of yours further proves you have never done a blind - you have to refer to Amir's blind testing for you examples, none of your own, I see. More & more absurd
Quote:
Me, I've only been debating about them with entities like you for a few decades now. Been a participant at Hydrogenaudio for ages. Even had beers with JJ once. And yes, I've done my own DBTs. I certainly don't care about yours.
Yep. lots of talk, I see but nothing of substance to back it up

Quote:
DBTs are, of course, used for interrogating preference as well...even when difference is 'gross'.

Btw, hope you didn't miss this in one of my other posts:
Do you really think there is something to miss in one of your posts?
jkeny is offline  
post #213 of 525 Old 01-26-2015, 01:17 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
Quote:
Originally Posted by krabapple View Post
Or put it this way: if you want to academically test the hypothesis "there an audible difference between A vs B", you of course do everything you can to maximize discriminatory power of the test, including training the subjects.

If you want to test some blowhard audiophile's claim that he routinely, already, hears a difference between A and B under condition X, you test that blowhard, under condition X....but blind.

So, in the trenches here, what do we encounter more often? Academic hypothesis, or someone's claim that they already hear a difference?
Yes, what's encountered here is the usual BS pseudo-science crowd sticking "blind test" labels to lego blocks & thinking they are doing real science.

Quote:
Yes, it's clear I've thought about these things more than you have. And I've been here 5 years longer.
Jeez, & that's what you came up with after 5 years of thinking? A shocking admission - I'm embarrassed for you!
jkeny is offline  
Sponsored Links
Advertisement
 
post #214 of 525 Old 01-26-2015, 01:18 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
It's called a hidden control & if you knew anything about blind testing you would be aware of it. from ITU document:
Indeed. But I was referring to something more amusing. Sometimes even inadvertant.


Quote:
Yes, all that is pre-test training - not done during the test trials, please. Every statement of yours further proves you have never done a blind - you have to refer to Amir's blind testing for you examples, none of your own, I see. More & more absurd Yep. lots of talk, I see but nothing of substance to back it up
I'm sure the audience is laughing.


Quote:
Do you really think there is something to miss in one of your posts?
Oops, well then, did I miss one of yours where you answered the question? (It was: jkeny, are you in 'the business'? (audio production, audio hardware/software design, audio sales, musician...?)
krabapple is offline  
post #215 of 525 Old 01-26-2015, 01:18 PM
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,829
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 1334 Post(s)
Liked: 751
Quote:
Originally Posted by jkeny View Post
And, even if expert, trained listeners submit to multiple tests, like Amir, you claim lying & cheating - really?
Indeed but there are steps within that. Here is how it goes:

1. You will never pass a level-matched DBT ABX.
2. You take and pass the level-matched DBT ABX that *they designed*. And one they are confident no one can pass
3. They come back to invalidate their own test. "Oh, there is a "big" problem with the test. Here is the fixed one."
4. You take the new test and pass that too.
5. They come and say, "well it must be because your system produces spurious response. Run this test and I am confident it will show your system to be broken." Answer is kept secret hoping you give the wrong one.
6. You take the test and report back the proper answer that indicates your system is just fine.
7. You ask them how come they are not taking the own test? They say their hearing is no good, or that they are afraid of ridicule with their negative outcome.
8. They say you must have cheated and hand modified the output of the Foobar program, or tested different files and just named them like they were supposed to be.
9. You re-run the tests again, this time with new version of Foobar ABX plug-in that computes a cryptographic hash (signature) of both the files and its output.
10. After you have done that, they come back and say that is no good because there are ways to still cheat and doctor up the output.
11. Ask them how that is done and whether they too can cheat and show positive results. They can't answer, nor do they produce said positive results by cheating.
12. They then ask for a live witness. As if any test they ever report has been audited by an independent person.
13. We finally get to your correct statement that where they leave it is, "it must have been a cheat or a lie."

It can't be the audio truth is different than their forum-built knowledge. It has to be that the person who did this work for a living, can show the credentials to be an expert listener, can pass every test thrown at him and run it precisely described, and has the technical knowledge to back what he is looking for, is a cheat.

Well no. The hat has been handed to them. But lack of common sense still entices them to boldly scream about double blind tests, the very thing they are distancing themselves from.
Charles R and PDRCanada like this.
amirm is offline  
post #216 of 525 Old 01-26-2015, 01:22 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
Yes, what's encountered here is the usual BS pseudo-science crowd sticking "blind test" labels to lego blocks & thinking they are doing real science.
'Blind test' labels on Lego blocks....? OK. That's a new one.

Quote:
Jeez, & that's what you came up with after 5 years of thinking? A shocking admission - I'm embarrassed for you!
I feel so naked.
krabapple is offline  
post #217 of 525 Old 01-26-2015, 01:30 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
So, let's just get this straight now - you disagree with spkr "No such thing as fatigue, loss of focus, etc., mate"

You, in fact think that it's tiring to find subtle differences in blind tests & even more difficult to try to find differences in the hidden controls used to detect false negatives & false positives. But hey, it's not important because blind tests should only be used for obvious differences. Maybe that's what spkr meant - blind tests should only be used for obvious differences & that's why "No such thing as fatigue, loss of focus, etc., mate"

Is this correct?
jkeny is offline  
post #218 of 525 Old 01-26-2015, 01:35 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by amirm View Post
Indeed but there are steps within that. Here is how it goes:

1. You will never pass a level-matched DBT ABX.
2. You take and pass the level-matched DBT ABX that *they designed*. And one they are confident no one can pass
3. They come back to invalidate their own test. "Oh, there is a "big" problem with the test. Here is the fixed one."
4. You take the new test and pass that too.
5. They come and say, "well it must be because your system produces spurious response. Run this test and I am confident it will show your system to be broken." Answer is kept secret hoping you give the wrong one.
6. You take the test and report back the proper answer that indicates your system is just fine.
7. You ask them how come they are not taking the own test? They say their hearing is no good, or that they are afraid of ridicule with their negative outcome.
8. They say you must have cheated and hand modified the output of the Foobar program, or tested different files and just named them like they were supposed to be.
9. You re-run the tests again, this time with new version of Foobar ABX plug-in that computes a cryptographic hash (signature) of both the files and its output.
10. After you have done that, they come back and say that is no good because there are ways to still cheat and doctor up the output.
11. Ask them how that is done and whether they too can cheat and show positive results. They can't answer, nor do they produce said positive results by cheating.
12. They then ask for a live witness. As if any test they ever report has been audited by an independent person.
13. We finally get to your correct statement that where they leave it is, "it must have been a cheat or a lie."

It can't be the audio truth is different than their forum-built knowledge. It has to be that the person who did this work for a living, can show the credentials to be an expert listener, can pass every test thrown at him and run it precisely described, and has the technical knowledge to back what he is looking for, is a cheat.

Well no. The hat has been handed to them. But lack of common sense still entices them to boldly scream about double blind tests, the very thing they are distancing themselves from.

I could have sworn it was you yourself who at least once said we can't consider self-reported results as really scientific (I would say 'dispositive'). And you are all about the *science* and the *rigor*, right?

Was I dreaming all that?

I think it's fun you can pass two amusingly and repeatedly and amateurishly bollixed-up tests of high-rez here on AVSF, and it's fun that Bob Stuart can arrange conditions under which *his* golden ear subjects (trained, or maybe not?) can (barely) hear filters, but I find it laughable to think that those differences are what Jason Serinus et al -- and please note that et al., in embraces *mulititudes* of testimonials and statements of certainty -- are reporting when they swoon over (for example) high rez.

You yourself (unless I was dreaming again), also have called these differences quite minor, hardly justifying the flowery verbiage spent on them. It always, always, always takes you awhile, Amir, but I will give you credit: if one has the patience to stick with you, and pesters you enough, you eventually do admit the important truth.
CruelInventions likes this.

Last edited by krabapple; 01-26-2015 at 01:45 PM.
krabapple is offline  
post #219 of 525 Old 01-26-2015, 01:40 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
Quote:
Originally Posted by amirm View Post
Indeed but there are steps within that. Here is how it goes:

1. You will never pass a level-matched DBT ABX.
2. You take and pass the level-matched DBT ABX that *they designed*. And one they are confident no one can pass
3. They come back to invalidate their own test. "Oh, there is a "big" problem with the test. Here is the fixed one."
4. You take the new test and pass that too.
5. They come and say, "well it must be because your system produces spurious response. Run this test and I am confident it will show your system to be broken." Answer is kept secret hoping you give the wrong one.
6. You take the test and report back the proper answer that indicates your system is just fine.
7. You ask them how come they are not taking the own test? They say their hearing is no good, or that they are afraid of ridicule with their negative outcome.
8. They say you must have cheated and hand modified the output of the Foobar program, or tested different files and just named them like they were supposed to be.
9. You re-run the tests again, this time with new version of Foobar ABX plug-in that computes a cryptographic hash (signature) of both the files and its output.
10. After you have done that, they come back and say that is no good because there are ways to still cheat and doctor up the output.
11. Ask them how that is done and whether they too can cheat and show positive results. They can't answer, nor do they produce said positive results by cheating.
12. They then ask for a live witness. As if any test they ever report has been audited by an independent person.
13. We finally get to your correct statement that where they leave it is, "it must have been a cheat or a lie."

It can't be the audio truth is different than their forum-built knowledge. It has to be that the person who did this work for a living, can show the credentials to be an expert listener, can pass every test thrown at him and run it precisely described, and has the technical knowledge to back what he is looking for, is a cheat.

Well no. The hat has been handed to them. But lack of common sense still entices them to boldly scream about double blind tests, the very thing they are distancing themselves from.
But with the display of logic (never mind common sense) on display here by this group, does it really surprise you?

Each step is met with more absurd logic:-
- I remember seeing it suggested that the "Live witness" had to be somebody they trusted :0
- And this one is a new one & a real doozy - now you have to "prove" that the difference you hear in the blind test is the same difference that you hear sighted.

As I said, it's a theatre of the absurd & anything goes.
Audionut11 and PDRCanada like this.
jkeny is offline  
post #220 of 525 Old 01-26-2015, 01:43 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
So, let's just get this straight now - you disagree with spkr "No such thing as fatigue, loss of focus, etc., mate"

You, in fact think that it's tiring to find subtle differences in blind tests & even more difficult to try to find differences in the hidden controls used to detect false negatives & false positives. But hey, it's not important because blind tests should only be used for obvious differences. Maybe that's what spkr meant - blind tests should only be used for obvious differences & that's why "No such thing as fatigue, loss of focus, etc., mate"

Is this correct?
Hey, I can do this too!

So, the only audio DBT result you will take seriously is one from a test that meets ITU spec, is this correct?
krabapple is offline  
post #221 of 525 Old 01-26-2015, 02:04 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
Quote:
Originally Posted by krabapple View Post
Hey, I can do this too!

So, the only audio DBT result you will take seriously is one from a test that meets ITU spec, is this correct?
I certainly will inform myself of what constitutes a valid test, any shortfall in meeting that criteria & make a judgement on that basis. But yes, I recognise something that you & spkr have demonstrated sails right over your heads - perceptual testing is a very difficult test & needs careful attention to detail & knowledge of the influencing factors.

Again, I will quote the ITU standard which you still haven't read:
Quote:
This Recommendation is intended for use in the assessment of systems which introduce impairments so small as to be undetectable without rigorous control of the experimental conditions and appropriate statistical analysis. If used for systems that introduce relatively large and easily detectable impairments, it leads to excessive expenditure of time and effort and may also lead to less reliable results than a simpler test. This Recommendation forms the base reference for the other Recommendations, which may contain additional special conditions or relaxations of the requirements included in this Recommendation
You & spkr demonstrate such an ignorance of the criteria for valid blind tests, never having read the documents (in your 5 years of thinking) & never having run a SBT that you have come up with your own set of illogical criteria
- "No such thing as fatigue, loss of focus, etc., mate"
- blind tests should only be used for obvious differences other wise you get tired
- training is done during listening trials
- no idea or use of hidden controls
- & the best of all - what is heard blind has to be proven to be what was heard sighted In your 5 years of thinking about this - how exactly do you propose to test this?

But hey when you are not interested in what is true & just want to drive an agenda, why would logic come into it? Any old test will do, once it delivers the null result & after all, it's DBT, it's "science" - sure doesn't that show science & measurements are correct & listening reports are just plain silly "night & day", "veil lifting" fantasies?

Last edited by jkeny; 01-26-2015 at 02:20 PM.
jkeny is offline  
post #222 of 525 Old 01-26-2015, 02:18 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
I certainly will inform myself of what constitutes a valid test, any shortfall in meeting that criteria & make a judgement on that basis.
Um...is that a yes? So, where do Amir's and your self-reported results fall on that spectrum of validity?


Quote:
But yes, I recognise something that you & spkr have demonstrated sails right over your heads - perceptual testing is a very difficult test & needs careful attention to detail & knowledge of the influencing factors.

Again, I will quote the ITU standard which you still haven't read:

I will add mind reader to your many obvious and subtle talents.


Meanwhile I['m enjoying a post by someone who disdains audio DBT even more than you do. I think he even pooh poohs the scientific ones. But I will assume you agree with him on everything, just because.

http://www.analogplanet.com/content/...ting-them-here

Last edited by krabapple; 01-26-2015 at 02:23 PM.
krabapple is offline  
post #223 of 525 Old 01-26-2015, 02:19 PM
FMW
AVS Forum Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 7,480
Mentioned: 13 Post(s)
Tagged: 1 Thread(s)
Quoted: 1654 Post(s)
Liked: 1710
Quote:
Originally Posted by jkeny View Post
But hey when you are not interested in what is true & just want to drive an agenda, why would logic come into it? Any old test will do, once it delivers the null result & after all, it's DBT, it's "science" - sure doesn't that show science & measurements are correct & listening reports are just plain silly "night & day", "veil lifting" fantasies?

Some audible differences disappear in blind tests. You can ignore that, if you like. You can call it nonsense. You can disbelieve it. But I'm not sure what sort of an agenda you are trying to gore in others.


Listening reports done with hearing bias about products shown not to have audible differences in blind tests are caused by hearing bias. To think otherwise is just plain irrational.


Why not get some experience in the matter yourself instead of throwing stones from outside the ropes? Are you afraid of something? What sort of anger motivates someone to attack people with personal experience when you have none yourself. It just doesn't make sense.


We've heard all these arguments before time and again like a broken record. It all relates to beliefs vs test results. Have your beliefs. It's fine with me.
FMW is offline  
post #224 of 525 Old 01-26-2015, 02:33 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
[quote=FMW;31186545]Some audible differences disappear in blind tests. You can ignore that, if you like. You can call it nonsense. You can disbelieve it. But I'm not sure what sort of an agenda you are trying to gore in others.[/qute]Many differences, not just subtle ones, will disappear in "blind tests" of the quality of what spkr & krabapple propose. Do you understand why there are such finely specified details contained in the standards & other documents for how these tests should be run & controlled? It's because testing perception is a very difficult task best left to research laboratories that have the money, expertise & staff to run them correctly.


Quote:
Listening reports done with hearing bias about products shown not to have audible differences in blind tests are caused by hearing bias. To think otherwise is just plain irrational.


Why not get some experience in the matter yourself instead of throwing stones from outside the ropes? Are you afraid of something? What sort of anger motivates someone to attack people with personal experience when you have none yourself. It just doesn't make sense.
Did you not see the positive ABX test results from me? Was I cheating? This is evidence that I ran a blind test - where is there any evidence that spkr & krabapple have ever done a blind test? Quite the contrary, they demonstrate a complete lack of knowledge of all aspects of blind tests


Quote:
We've heard all these arguments before time and again like a broken record. It all relates to beliefs vs test results. Have your beliefs. It's fine with me.
No need to read the posts then - your choice!
jkeny is offline  
post #225 of 525 Old 01-26-2015, 02:39 PM
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,829
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 1334 Post(s)
Liked: 751
Quote:
Originally Posted by krabapple View Post
I could have sworn it was you yourself who at least once said we can't consider self-reported results as really scientific (I would say 'dispositive').
Not at all. For the purposes of having a discussion on a forum, reporting such results are useful. I am sure you remember recently when Arny post his results: https://www.avsforum.com/forum/91-aud...l#post30565497

Quote:
Originally Posted by amirm View Post
Quote:
Originally Posted by arnyk with Amir's addition of time lapse
21:04:53 : Test started.
21:05:18 : 00/01
21:05:39 : 01/02 --- 21 seconds
21:06:39 : 02/03 --- 60 seconds
21:06:45 : 03/04 --- 6 seconds
21:06:47 : 04/05 --- 2 seconds!!!
21:06:50 : 04/06 --- 3 seconds!!!
21:06:54 : 04/07 --- 4 seconds!!!
21:06:56 : 05/08 --- 2 seconds!!!
21:06:58 : 06/09 --- 2 seconds!!!
21:06:59 : 07/10 --- 1 second!!!!
21:07:01 : 07/11 --- 2 seconds!!!
21:07:04 : 07/12 --- 3 seconds!!!
21:07:05 : 08/13 --- 1 second!!!
21:07:08 : 08/14 --- 3 seconds!!!
21:07:10 : 08/15 --- 2 seconds!!!
21:07:31 : 08/16 --- 21 seconds
21:07:31 : Test finished.

----------
Total: 8/16
Probability that you were guessing: 59.8%
Thank you Arny. I know you don't realize it but there is tremendous learning in what you post.

I took the liberty of adding the time it took for you to vote on each iteration and hitting "next trial." We see that when you first started (in blue), you were making an effort to tell the clips apart. You took 21 seconds followed by 60 seconds. The 60 seconds is quite long. Clearly you decided you could not which is fine. But here is the learning: you did not stop! You kept going and voted immediately with some random outcome as noted by the time periods in red. Since it takes a second or two just to move the mouse and hit "Next Trial," You are literally voting randomly as fast as you possibly can. Clearly you were not even listening the clips anymore. Just voting to get the test done.

Now let's say you and I were both taking this test and our results were combined. All of my solid results would then be diluted by your random voting. You were not even listening to the clips yet someone just blindly, pun intended, adding up the scores would count you as making a valid effort.

This is why proper tests of this sort screen out listeners like yourself. Positive controls are included to detect someone just voting without listening. You would have been found out and excluded from the formal testing.

Your results also shows how easy it is to produce negative results in forced choice tests like ABX (i.e. both answers are available to you). Just close your eyes and hit the buttons randomly. Not so when it comes to positive results. So to the extent the listener is put in a situation of taking a difficult test, you are almost assured of negative outcome, the real difference that may exist notwithstanding.
So clearly I welcome such testing as it nicely shows the issues with execution of such tests, even if it comes from self-claimed experts in this area.

What I am not in favor of and might be what you are thinking is joe-forum-guy thinking he has created science by running some random test and claiming that is the way the audio world turns.

Quote:
And you are all about the *science* and the *rigor*, right?
I am. Are you saying you don't know how to run a double blind test with rigor and report?
amirm is offline  
post #226 of 525 Old 01-26-2015, 02:46 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by amirm View Post
Not at all. For the purposes of having a discussion on a forum, reporting such results are useful.
OK, lovely sentiment,, but not quite pertinent to what I wrote,which was:

Quote:
I could have sworn it was you yourself who at least once said we can't consider self-reported results as really scientific
Did I imagine that?

If not, then, your current stance is, reporting such results is useful, even if not scientific? But by 'useful', you mean, useful for showing how bad unscientific DBTs are?
krabapple is offline  
post #227 of 525 Old 01-26-2015, 02:48 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by FMW View Post
We've heard all these arguments before time and again like a broken record. It all relates to beliefs vs test results. Have your beliefs. It's fine with me.

Yeah, I think most of the audience has left the house.
krabapple is offline  
post #228 of 525 Old 01-26-2015, 02:50 PM
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,829
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 1334 Post(s)
Liked: 751
Quote:
Originally Posted by krabapple View Post
I think it's fun you can pass two amusingly and repeatedly and amateurishly bollixed-up tests of high-rez here on AVSF..
Two tests? You are way, way wrong in your score keeping. Here are the tests I have run with positive outcomes:

1. Scott/Mark high-res vs CD test in the sticky on this subforum. That test had three (or was it four) tracks. I passed all.
2. Arny's jitter tests.
3. Arny's High-res vs. CD test.
4. 320 kbps MP3 vs. Scott/Mark clip per your claim that I could not.
5. 320 kbps MP3 vs. Nordic Sampler that Arny used and I quoted in my last response.
4. Ethan's generational test (looping content in and out of a DAC/ADC multiple times).
5. Archimago test of 16 vs 24 bits.
6. David's test of filtering high-resolution audio on Hydrogenaudio forum

This is all that I remember just now. And it is not that I passed these tests but that I passed them shortly after demand that I do so. In all cases the tests and conditions were created by my antagonists hoping that I could not pass.

All in all I have spent countless hours running them. Here is a key message: we are telling the world that they better not take any such tests. Because if they do, at the end of test after test, you are simply accused of being a cheater and liar. If you are in favor of people running more of these tests, then you sure have created a situation where no one would.

Of course the fact that none of you have run any of these tests puts folks who refuse in your company. So there better be no more complaining about people not running double blind tests.

Anyone in favor of people running such tests would be the first to volunteer to run them, not be afraid of any outcome, and welcoming of the others running them. But you are not. Why? Because you are not on the side of right and you now know it.
Charles R and PDRCanada like this.
amirm is offline  
post #229 of 525 Old 01-26-2015, 02:56 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by jkeny View Post
Quote:
Originally Posted by FMW View Post
Some audible differences disappear in blind tests. You can ignore that, if you like. You can call it nonsense. You can disbelieve it. But I'm not sure what sort of an agenda you are trying to gore in others.
Many differences, not just subtle ones, will disappear in "blind tests" of the quality of what spkr & krabapple propose.

Funny thing is when : 1) the listener goes into the DBT claiming they already hear a difference between the A and B under test, and 2) they continue to 'hear' the difference during the test. But the results indicate a score no better than chance. (or funnier still, an alternate (2): the subject reports that A and B became *much harder to tell apart* under blind conditions, than when they were 'seen')

How does that happen?
krabapple is offline  
post #230 of 525 Old 01-26-2015, 03:02 PM
AVS Forum Special Member
 
krabapple's Avatar
 
Join Date: Apr 2003
Location: in a state bordered by Kentucky and Maine
Posts: 5,779
Mentioned: 3 Post(s)
Tagged: 0 Thread(s)
Quoted: 487 Post(s)
Liked: 469
Quote:
Originally Posted by amirm View Post
Two tests? You are way, way wrong in your score keeping. Here are the tests I have run with positive outcomes:

1. Scott/Mark high-res vs CD test in the sticky on this subforum. That test had three (or was it four) tracks. I passed all.
2. Arny's jitter tests.
3. Arny's High-res vs. CD test.
4. 320 kbps MP3 vs. Scott/Mark clip per your claim that I could not.
5. 320 kbps MP3 vs. Nordic Sampler that Arny used and I quoted in my last response.
4. Ethan's generational test (looping content in and out of a DAC/ADC multiple times).
5. Archimago test of 16 vs 24 bits.
6. David's test of filtering high-resolution audio on Hydrogenaudio forum

...and I specifically wrote 'here on AVSF'. I was thinking of the AIX test and Arny's keys. I can't say I've followed your other exploits as closely as you'd like, alas.. Your need for attention is quite exhausting!


Quote:
This is all that I remember just now. And it is not that I passed these tests but that I passed them shortly after demand that I do so. In all cases the tests and conditions were created by my antagonists hoping that I could not pass.
But IIRC the point you made in at least the few I did follow, was that you are extra-special-super-secret-agent-listener-trained? (And also, that the differences were very small/required homing in on a fleeting 'tell'..which, last time I bothered to pay close attention, you had not yet revealed) OK, if we accept that, what are we to make of the sorts of claims we encounter over countless hours from audiophiles, not to mention 'the industry'? Is it likely they are all being made by ESSSALSTs as well?
krabapple is offline  
post #231 of 525 Old 01-26-2015, 03:25 PM
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,829
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 1334 Post(s)
Liked: 751
Quote:
Originally Posted by krabapple View Post
...and it's fun that Bob Stuart can arrange conditions under which *his* golden ear subjects (trained, or maybe not?) can (barely) hear filters,...
Sorry, no. That is wrong. Here is the paper:

The audibility of typical digital audio lters in a high-delity playback system
Helen M. Jackson, Michael D. Capp, and J. Robert Stuart

"All forms of processing tested here were audi-
ble
[conversion from 24 bit/192 Khz to 44/48 Khz at 24 and 16 bits],
except for one condition where performance was
signicantly dierent from chance at the 6.7% level,
including emulated downsampling lters at standard
sample rates and 16-bit quantization with or without
RPDF dither. Differences were demonstrated here
in a double-blind test using non-expert listeners who
received minimal training.
"


jkeny is right. You don't seem to be familiar at all with the protocol and terminology used in double blind tests. Yes there are a lot of references to training in the paper/test but that is not the same as one being a golden ear subject. A listener must be trained to know how to take the test properly, and have opportunity to become familiar with the set up, the content, what differences may be, etc. We do not want them to be dumb and stupid machines and just push buttons. We want to extract knowledge so anything we can do to guide a test subject to find the differences that are there, must be done.

Being an expert/golden ear listener is very different. That is a process of going through extensive training across a ton of tests, learning the algorithms in question, participating in many tests, interacting with other experts, formal training, etc. It took me some 6 months to finish this process and come out the other end as a trained listener.

It is like you learning to play a few note on a Piano from an expert. That one session does not make you a Piano player and certainly no expert.

There are basic, basic things about how to do the tests properly. But they are foreign to people online who think the only qualification to take a test is to not be deaf as FMW wrongly said.

Quote:
..but I find it laughable to think that those differences are what Jason Serinus et al -- and please note that et al., in embraces *mulititudes* of testimonials and statements of certainty -- are reporting when they swoon over (for example) high rez.
Don't know what this stream of consciousness means. The test set out to disprove the position you all hold and it did so in spades. It won an award for best peer reviewed conference paper. No Joe random forum poster gets to reposition it as something else, when even the vocabulary of conducting double blind tests is not understood.

"Laughable?" Unbelievable. Cry should be the operative word if I were in your shoes...
Charles R likes this.
amirm is offline  
post #232 of 525 Old 01-26-2015, 03:41 PM
 
amirm's Avatar
 
Join Date: Jan 2002
Location: Washington State
Posts: 18,829
Mentioned: 6 Post(s)
Tagged: 0 Thread(s)
Quoted: 1334 Post(s)
Liked: 751
Quote:
Originally Posted by krabapple View Post
...and I specifically wrote 'here on AVSF'. I was thinking of the AIX test and Arny's keys. I can't say I've followed your other exploits as closely as you'd like, alas.. Your need for attention is quite exhausting!
If it is exhausting, you may want to refrain from manufacturing things. It certainly is exhausting for me to keep responding to incorrect things you are saying in these posts. Here is yet another example in the form of Ethan's test that I said I ran: https://www.avsforum.com/forum/91-aud...l#post25869354

Quote:
Originally Posted by amirm View Post
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/18 06:40:07

File A: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_original.wav
File B: C:\Users\Amir\Music\Ethan Soundblaster\sb20x_pass1.wav

06:40:07 : Test started.
06:41:03 : 01/01 50.0%
06:41:16 : 02/02 25.0%
06:41:24 : 03/03 12.5%
06:41:33 : 04/04 6.3%
06:41:53 : 05/05 3.1%
06:42:02 : 06/06 1.6%
06:42:22 : 07/07 0.8%
06:42:34 : 08/08 0.4%
06:42:43 : 09/09 0.2%
06:42:56 : 10/10 0.1%
06:43:08 : 11/11 0.0%
06:43:16 : Test finished.

----------
Total: 11/11 (0.0%)

===========
I don't think I failed .
Just look a few posts up and you see me responding to your post. So you read about this other test.

Here is me in that very same thread responding to you implying I can't pass a double blind test of 320 kbps MP3 against the original: https://www.avsforum.com/forum/91-aud...l#post26197034

Quote:
Originally Posted by amirm View Post
Quote:
Originally Posted by Krabapple

And then the dancers will do their dance to convince them they're right. I imagine some will point to this very thread!
To wit: Is the Dancing Man from Madrona really now saying he routinely discerns 320 kbps MP3s from source?
I didn't say that. But I do OK. As a random challenge, I converted Arny's clip to MP3 at 320 kbps a few days ago and post the results in another thread on AVS:

=============
foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/19 19:45:33

File A: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44.wav
File B: C:\Users\Amir\Music\Arnys Filter Test\keys jangling 16 44_01.mp3

19:45:33 : Test started.
19:46:21 : 01/01 50.0%
19:46:35 : 02/02 25.0%
19:46:49 : 02/03 50.0%
19:47:03 : 03/04 31.3%
19:47:13 : 04/05 18.8%
19:47:27 : 05/06 10.9%
19:47:38 : 06/07 6.3%
19:47:46 : 07/08 3.5%
19:48:01 : 08/09 2.0%
19:48:19 : 09/10 1.1%
19:48:31 : 10/11 0.6%
19:48:45 : 11/12 0.3%
19:48:58 : 12/13 0.2%
19:49:11 : 13/14 0.1%
19:49:28 : 14/15 0.0%
19:49:52 : 15/16 0.0%
19:49:56 : Test finished.

----------
Total: 15/16 (0.0%)

And just now, ran one of the tracks from the 6 samples in Scott/Mark's music and converted that to MP3. Here are those results:

===============

foo_abx 1.3.4 report
foobar2000 v1.3.2
2014/07/31 15:18:41

File A: C:\Users\Amir\Music\AIX AVS Test files\On_The_Street_Where_You_Live_A2.mp3
File B: C:\Users\Amir\Music\AIX AVS Test files\On_The_Street_Where_You_Live_A2.wav

15:18:41 : Test started.
15:19:18 : 01/01 50.0%
15:19:30 : 01/02 75.0%
15:19:44 : 01/03 87.5%
15:20:35 : 02/04 68.8%
15:20:46 : 02/05 81.3%
15:21:39 : 03/06 65.6% <--- Difference found
15:21:47 : 04/07 50.0%
15:21:54 : 04/08 63.7% <--- Dog barked!
15:22:06 : 05/09 50.0%
15:22:19 : 06/10 37.7%
15:22:31 : 07/11 27.4%
15:22:44 : 08/12 19.4%
15:22:51 : 09/13 13.3%
15:22:58 : 10/14 9.0%
15:23:06 : 11/15 5.9%
15:23:14 : 12/16 3.8%
15:23:23 : 13/17 2.5%
15:23:33 : 14/18 1.5%
15:23:42 : 15/19 1.0%
15:23:54 : 16/20 0.6%
15:24:06 : 17/21 0.4%
15:24:15 : 18/22 0.2%
15:24:23 : 19/23 0.1%
15:24:34 : 20/24 0.1%
15:24:43 : 21/25 0.0%
15:24:52 : 22/26 0.0%
15:24:57 : Test finished.

----------
Total: 22/26 (0.0%)

As you see, the results could not be more compelling on me being able to tell 320 kbps apart from the source. And these are not tracks selected to be "codec killers." It is what folks have put forward in this forum for another reason.

So that folks don't stay upset at me, talking to you Chu, I saved the markers for the above track:

Start point: 3:11.4
End point: 3:11.8

Listen to the difference in high frequencies. Listen to whether one is more congested than the other (for the lack of a better word). Now you see why I don't try to explain the differences. English words are not easy fit for them. This is why you need to do your own listening and hear the difference first hand. No amount of telling you how to fish will teach you what it feels like to catch your first one.
All on AVS. All in front of you.

You attempted to belittle the results of these positive outcomes and I am here to correct you. Stick with the facts and real data and I won't post. It is that simple.
Charles R and PDRCanada like this.
amirm is offline  
post #233 of 525 Old 01-26-2015, 04:01 PM
Member
 
jkeny's Avatar
 
Join Date: Nov 2007
Posts: 74
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 71 Post(s)
Liked: 23
Quote:
Originally Posted by krabapple View Post
Funny thing is when : 1) the listener goes into the DBT claiming they already hear a difference between the A and B under test, and 2) they continue to 'hear' the difference during the test. But the results indicate a score no better than chance. (or funnier still, an alternate (2): the subject reports that A and B became *much harder to tell apart* under blind conditions, than when they were 'seen')

How does that happen?
You will never know given your test methods. Anyway, I don't believe you are interested in answering this.

Last edited by jkeny; 01-26-2015 at 04:27 PM.
jkeny is offline  
post #234 of 525 Old 01-26-2015, 05:00 PM
AVS Forum Special Member
 
Randy Bessinger's Avatar
 
Join Date: Jun 2009
Posts: 1,877
Mentioned: 85 Post(s)
Tagged: 0 Thread(s)
Quoted: 353 Post(s)
Liked: 333

Posted by Desertdome in the kc meet thread. The kc guys have done a number of blind and DBT tests
Tack likes this.
Randy Bessinger is offline  
post #235 of 525 Old 01-26-2015, 05:36 PM
AVS Forum Addicted Member
 
Tack's Avatar
 
Join Date: Mar 2004
Location: Valencia, Ca
Posts: 11,280
Mentioned: 10 Post(s)
Tagged: 0 Thread(s)
Quoted: 4840 Post(s)
Liked: 6507
Quote:
Originally Posted by Randy Bessinger View Post

Posted by Desertdome in the kc meet thread. The kc guys have done a number of blind and DBT tests
That's nicely done. The pacing is perfect.
Tack is offline  
post #236 of 525 Old 01-26-2015, 08:55 PM - Thread Starter
 
NorthSky's Avatar
 
Join Date: Jan 2013
Location: Star of the Northern Hemisphere
Posts: 16,643
Mentioned: 26 Post(s)
Tagged: 0 Thread(s)
Quoted: 7012 Post(s)
Liked: 3562
Quote:
Originally Posted by krabapple View Post
Yeah, I think most of the audience has left the house.
Nope, they're all still here, reading and learning. ...When they'll have smart questions, they'll know exactly where to ask them and to who.
NorthSky is offline  
post #237 of 525 Old 01-26-2015, 10:29 PM
 
F.Cook's Avatar
 
Join Date: Jan 2015
Location: Ontario
Posts: 282
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 152 Post(s)
Liked: 67
Quote:
Originally Posted by NorthSky View Post
Just a simple short link people can read, and comment on. ...If they wish. ...Me, I found it quite captivating.

http://seanolive.blogspot.ie/2009/04...o-product.html
In that the context of the cited article is based on speaker evaluation, a few things come to mind:


1. All speaker tests, are cited evaluations. There is no retail outlet, that offers properly controlled DB ABX tests of any kind, let alone speakers. Making DB ABX comparisons moot.


2. The Sonic differences between speakers, has been well evidenced as being easily discernable, by even amateur observers, visa vie DB ABX test performed in years gone by.


3. Sight evaluations of CD Transports, DAC, and Amplifiers and Preamps, is a different story, as most DB ABX tests have produced null's as outcomes; however, Sighted evaluations have produced weighted outcomes...


4. Sighted evaluations which were performed with the equipment cover up, have produced similar results to that of DB ABX tests (not including speakers).


5. Sighted evaluations are arguably, the only metric available to the masses, and I would argue relentlessly that 99% of AVS members and onlookers within this forum, have made their buying decision visa vie sighted evaluations, and that furthermore, that the loins share is completely satisfied with their decisions(s)!


6. BD ABX testing is extremely complex to proctor, on many levels... I don't know of a single QSC comparator in use and available to even a smallish portion of the population. Only conjecture can exist in the current landscape. But again, empirically, sighted evaluations have resulted in hundreds of millions of satisfied customers.


7. I don't believe that their bad, poor or any such thing - they're certainly not dishonest! Removing sighted evaluations, and leaving only the verbal regaling's of sales personnel to depict the performance attributes of products, would absolutely be a dishonest evaluation metric. After all, by what means would the sale staff garner their opinions? Please don't say technical specifications. And we all know that it wouldn't be visa vie BD ABX testing's...


8. Sighted evaluations are simply all that's within a scope of practical implementation and therefore reach. Without them, we would truly be making a blind decision.

Last edited by F.Cook; 01-26-2015 at 10:34 PM.
F.Cook is offline  
post #238 of 525 Old 01-26-2015, 11:01 PM - Thread Starter
 
NorthSky's Avatar
 
Join Date: Jan 2013
Location: Star of the Northern Hemisphere
Posts: 16,643
Mentioned: 26 Post(s)
Tagged: 0 Thread(s)
Quoted: 7012 Post(s)
Liked: 3562
I love your eight points above, in particular the last one.

In the link contained in the quoting post above (original first post), on Sean's blog Alan Sircom, pro audio reviewer from the UK - Hi-Fi+ -, and also a member over @ WBF (and most likely here too @ AVSForum), occasionally posting intelligently, had a very realistic comment for Dr. Sean Olive; it's worth double-checking, and Sean's reply.

____________


Nice to see new faces around with well spoken facts.
NorthSky is offline  
post #239 of 525 Old 01-27-2015, 12:22 AM
FMW
AVS Forum Special Member
 
FMW's Avatar
 
Join Date: Jun 2007
Posts: 7,480
Mentioned: 13 Post(s)
Tagged: 1 Thread(s)
Quoted: 1654 Post(s)
Liked: 1710
Quote:
Originally Posted by F.Cook View Post
In that the context of the cited article is based on speaker evaluation, a few things come to mind:


1. All speaker tests, are cited evaluations. There is no retail outlet, that offers properly controlled DB ABX tests of any kind, let alone speakers. Making DB ABX comparisons moot.


2. The Sonic differences between speakers, has been well evidenced as being easily discernable, by even amateur observers, visa vie DB ABX test performed in years gone by.


3. Sight evaluations of CD Transports, DAC, and Amplifiers and Preamps, is a different story, as most DB ABX tests have produced null's as outcomes; however, Sighted evaluations have produced weighted outcomes...


4. Sighted evaluations which were performed with the equipment cover up, have produced similar results to that of DB ABX tests (not including speakers).


5. Sighted evaluations are arguably, the only metric available to the masses, and I would argue relentlessly that 99% of AVS members and onlookers within this forum, have made their buying decision visa vie sighted evaluations, and that furthermore, that the loins share is completely satisfied with their decisions(s)!


6. BD ABX testing is extremely complex to proctor, on many levels... I don't know of a single QSC comparator in use and available to even a smallish portion of the population. Only conjecture can exist in the current landscape. But again, empirically, sighted evaluations have resulted in hundreds of millions of satisfied customers.


7. I don't believe that their bad, poor or any such thing - they're certainly not dishonest! Removing sighted evaluations, and leaving only the verbal regaling's of sales personnel to depict the performance attributes of products, would absolutely be a dishonest evaluation metric. After all, by what means would the sale staff garner their opinions? Please don't say technical specifications. And we all know that it wouldn't be visa vie BD ABX testing's...


8. Sighted evaluations are simply all that's within a scope of practical implementation and therefore reach. Without them, we would truly be making a blind decision.

I agree with all of your points except for #7 . Normally, a sighted evaluation will produce exactly the results that the salesperson suggested so that makes it dishonest in my book. I think there might be some value in sighted evaluations in rare cases but those cases would normally involve people who have no expectations. I can't imagine that it would involve anybody reading this thread and that is our audience. The salesman's comments and the sighted evaluation are the same thing in most cases.


What is a consumer to do? Choose what sounds best in whatever way it is possible to compare. But the informed consumer will do that with some knowledge of what blind tests have shown. As an example, I would never buy an external DAC for any reason other than for unusual switching and connection capabilities because I've been through the tests and I know better.
FMW is offline  
post #240 of 525 Old 01-27-2015, 01:24 AM
Advanced Member
 
jj_0001's Avatar
 
Join Date: Sep 2006
Location: In the rain
Posts: 888
Mentioned: 0 Post(s)
Tagged: 0 Thread(s)
Quoted: 152 Post(s)
Liked: 67
Quote:
Originally Posted by 8mile13 View Post
When you do a audio (blind)test sound is throwed in your face.
No, it's not "thrown in your face", you have plenty of time to listen to the sound, etc.

I have no idea what you're even imagining here, it sounds like some gross corruption of how any reasonable test should be run.

James D. (jj) Johnston
jj_0001 is offline  
Sponsored Links
Advertisement
 
Closed Thread Audio Theory, Setup, and Chat

Thread Tools
Show Printable Version Show Printable Version
Email this Page Email this Page


Forum Jump: 

Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off