Well, one never knows, does one?

It's called a hidden control & if you knew anything about blind testing you would be aware of it. from ITU document:
Yes, all that is pre-test training - not done during the test trials, please. Every statement of yours further proves you have never done a blind - you have to refer to Amir's blind testing for you examples, none of your own, I see. More & more absurd Yep. lots of talk, I see but nothing of substance to back it up

Do you really think there is something to miss in one of your posts?

Yes, what's encountered here is the usual BS pseudo-science crowd sticking "blind test" labels to lego blocks & thinking they are doing real science.

Jeez, & that's what you came up with after 5 years of thinking? A shocking admission - I'm embarrassed for you!

Indeed. But I was referring to something more amusing. Sometimes even inadvertant.


I'm sure the audience is laughing.


Oops, well then, did I miss one of yours where you answered the question? (It was: jkeny, are you in 'the business'? (audio production, audio hardware/software design, audio sales, musician...?)

Indeed but there are steps within that. Here is how it goes:

1. You will never pass a level-matched DBT ABX.
2. You take and pass the level-matched DBT ABX that *they designed*. And one they are confident no one can pass
3. They come back to invalidate their own test. "Oh, there is a "big" problem with the test. Here is the fixed one."
4. You take the new test and pass that too.
5. They come and say, "well it must be because your system produces spurious response. Run this test and I am confident it will show your system to be broken." Answer is kept secret hoping you give the wrong one.
6. You take the test and report back the proper answer that indicates your system is just fine.
7. You ask them how come they are not taking the own test? They say their hearing is no good, or that they are afraid of ridicule with their negative outcome.
8. They say you must have cheated and hand modified the output of the Foobar program, or tested different files and just named them like they were supposed to be.
9. You re-run the tests again, this time with new version of Foobar ABX plug-in that computes a cryptographic hash (signature) of both the files and its output.
10. After you have done that, they come back and say that is no good because there are ways to still cheat and doctor up the output.
11. Ask them how that is done and whether they too can cheat and show positive results. They can't answer, nor do they produce said positive results by cheating.
12. They then ask for a live witness. As if any test they ever report has been audited by an independent person.
13. We finally get to your correct statement that where they leave it is, "it must have been a cheat or a lie."

It can't be the audio truth is different than their forum-built knowledge. It has to be that the person who did this work for a living, can show the credentials to be an expert listener, can pass every test thrown at him and run it precisely described, and has the technical knowledge to back what he is looking for, is a cheat.

Well no. The hat has been handed to them. But lack of common sense still entices them to boldly scream about double blind tests, the very thing they are distancing themselves from.

'Blind test' labels on Lego blocks....? OK. That's a new one.

I feel so naked.

So, let's just get this straight now - you disagree with spkr "No such thing as fatigue, loss of focus, etc., mate"

You, in fact think that it's tiring to find subtle differences in blind tests & even more difficult to try to find differences in the hidden controls used to detect false negatives & false positives. But hey, it's not important because blind tests should only be used for obvious differences. Maybe that's what spkr meant - blind tests should only be used for obvious differences & that's why "No such thing as fatigue, loss of focus, etc., mate"

Is this correct?

I could have sworn it was you yourself who at least once said we can't consider self-reported results as really scientific (I would say 'dispositive'). And you are all about the *science* and the *rigor*, right?

Was I dreaming all that?

I think it's fun you can pass two amusingly and repeatedly and amateurishly bollixed-up tests of high-rez here on AVSF, and it's fun that Bob Stuart can arrange conditions under which *his* golden ear subjects (trained, or maybe not?) can (barely) hear filters, but I find it laughable to think that those differences are what Jason Serinus et al -- and please note that et al., in embraces *mulititudes* of testimonials and statements of certainty -- are reporting when they swoon over (for example) high rez.

You yourself (unless I was dreaming again), also have called these differences quite minor, hardly justifying the flowery verbiage spent on them. It always, always, always takes you awhile, Amir, but I will give you credit: if one has the patience to stick with you, and pesters you enough, you eventually do admit the important truth.

But with the display of logic (never mind common sense) on display here by this group, does it really surprise you?

Each step is met with more absurd logic:-
- I remember seeing it suggested that the "Live witness" had to be somebody they trusted :0
- And this one is a new one & a real doozy - now you have to "prove" that the difference you hear in the blind test is the same difference that you hear sighted.

As I said, it's a theatre of the absurd & anything goes.

Hey, I can do this too!

So, the only audio DBT result you will take seriously is one from a test that meets ITU spec, is this correct?

I certainly will inform myself of what constitutes a valid test, any shortfall in meeting that criteria & make a judgement on that basis. But yes, I recognise something that you & spkr have demonstrated sails right over your heads - perceptual testing is a very difficult test & needs careful attention to detail & knowledge of the influencing factors.

Again, I will quote the ITU standard which you still haven't read:
You & spkr demonstrate such an ignorance of the criteria for valid blind tests, never having read the documents (in your 5 years of thinking) & never having run a SBT that you have come up with your own set of illogical criteria
- "No such thing as fatigue, loss of focus, etc., mate"
- blind tests should only be used for obvious differences other wise you get tired
- training is done during listening trials
- no idea or use of hidden controls
- & the best of all - what is heard blind has to be proven to be what was heard sighted In your 5 years of thinking about this - how exactly do you propose to test this?

But hey when you are not interested in what is true & just want to drive an agenda, why would logic come into it? Any old test will do, once it delivers the null result & after all, it's DBT, it's "science" - sure doesn't that show science & measurements are correct & listening reports are just plain silly "night & day", "veil lifting" fantasies?

Um...is that a yes? So, where do Amir's and your self-reported results fall on that spectrum of validity?



I will add mind reader to your many obvious and subtle talents.


Meanwhile I['m enjoying a post by someone who disdains audio DBT even more than you do. I think he even pooh poohs the scientific ones. But I will assume you agree with him on everything, just because.

http://www.analogplanet.com/content/...ting-them-here

Some audible differences disappear in blind tests. You can ignore that, if you like. You can call it nonsense. You can disbelieve it. But I'm not sure what sort of an agenda you are trying to gore in others.


Listening reports done with hearing bias about products shown not to have audible differences in blind tests are caused by hearing bias. To think otherwise is just plain irrational.


Why not get some experience in the matter yourself instead of throwing stones from outside the ropes? Are you afraid of something? What sort of anger motivates someone to attack people with personal experience when you have none yourself. It just doesn't make sense.


We've heard all these arguments before time and again like a broken record. It all relates to beliefs vs test results. Have your beliefs. It's fine with me.

[quote=FMW;31186545]Some audible differences disappear in blind tests. You can ignore that, if you like. You can call it nonsense. You can disbelieve it. But I'm not sure what sort of an agenda you are trying to gore in others.[/qute]Many differences, not just subtle ones, will disappear in "blind tests" of the quality of what spkr & krabapple propose. Do you understand why there are such finely specified details contained in the standards & other documents for how these tests should be run & controlled? It's because testing perception is a very difficult task best left to research laboratories that have the money, expertise & staff to run them correctly.


Did you not see the positive ABX test results from me? Was I cheating? This is evidence that I ran a blind test - where is there any evidence that spkr & krabapple have ever done a blind test? Quite the contrary, they demonstrate a complete lack of knowledge of all aspects of blind tests


No need to read the posts then - your choice!

Not at all. For the purposes of having a discussion on a forum, reporting such results are useful. I am sure you remember recently when Arny post his results: https://www.avsforum.com/forum/91-aud...l#post30565497

So clearly I welcome such testing as it nicely shows the issues with execution of such tests, even if it comes from self-claimed experts in this area.

What I am not in favor of and might be what you are thinking is joe-forum-guy thinking he has created science by running some random test and claiming that is the way the audio world turns.

I am. Are you saying you don't know how to run a double blind test with rigor and report?

OK, lovely sentiment,, but not quite pertinent to what I wrote,which was:

Did I imagine that?

If not, then, your current stance is, reporting such results is useful, even if not scientific? But by 'useful', you mean, useful for showing how bad unscientific DBTs are?

Yeah, I think most of the audience has left the house.

Two tests? You are way, way wrong in your score keeping. Here are the tests I have run with positive outcomes:

1. Scott/Mark high-res vs CD test in the sticky on this subforum. That test had three (or was it four) tracks. I passed all.
2. Arny's jitter tests.
3. Arny's High-res vs. CD test.
4. 320 kbps MP3 vs. Scott/Mark clip per your claim that I could not.
5. 320 kbps MP3 vs. Nordic Sampler that Arny used and I quoted in my last response.
4. Ethan's generational test (looping content in and out of a DAC/ADC multiple times).
5. Archimago test of 16 vs 24 bits.
6. David's test of filtering high-resolution audio on Hydrogenaudio forum

This is all that I remember just now. And it is not that I passed these tests but that I passed them shortly after demand that I do so. In all cases the tests and conditions were created by my antagonists hoping that I could not pass.

All in all I have spent countless hours running them. Here is a key message: we are telling the world that they better not take any such tests. Because if they do, at the end of test after test, you are simply accused of being a cheater and liar. If you are in favor of people running more of these tests, then you sure have created a situation where no one would.

Of course the fact that none of you have run any of these tests puts folks who refuse in your company. So there better be no more complaining about people not running double blind tests.

Anyone in favor of people running such tests would be the first to volunteer to run them, not be afraid of any outcome, and welcoming of the others running them. But you are not. Why? Because you are not on the side of right and you now know it.

Funny thing is when : 1) the listener goes into the DBT claiming they already hear a difference between the A and B under test, and 2) they continue to 'hear' the difference during the test. But the results indicate a score no better than chance. (or funnier still, an alternate (2): the subject reports that A and B became *much harder to tell apart* under blind conditions, than when they were 'seen')

How does that happen?

...and I specifically wrote 'here on AVSF'. I was thinking of the AIX test and Arny's keys. I can't say I've followed your other exploits as closely as you'd like, alas.. Your need for attention is quite exhausting!


But IIRC the point you made in at least the few I did follow, was that you are extra-special-super-secret-agent-listener-trained? (And also, that the differences were very small/required homing in on a fleeting 'tell'..which, last time I bothered to pay close attention, you had not yet revealed) OK, if we accept that, what are we to make of the sorts of claims we encounter over countless hours from audiophiles, not to mention 'the industry'? Is it likely they are all being made by ESSSALSTs as well?

Sorry, no. That is wrong. Here is the paper:

The audibility of typical digital audio lters in a high-delity playback system
Helen M. Jackson, Michael D. Capp, and J. Robert Stuart

"All forms of processing tested here were audi-
ble [conversion from 24 bit/192 Khz to 44/48 Khz at 24 and 16 bits],
except for one condition where performance was
signicantly dierent from chance at the 6.7% level,
including emulated downsampling lters at standard
sample rates and 16-bit quantization with or without
RPDF dither. Differences were demonstrated here
in a double-blind test using non-expert listeners who
received minimal training."

jkeny is right. You don't seem to be familiar at all with the protocol and terminology used in double blind tests. Yes there are a lot of references to training in the paper/test but that is not the same as one being a golden ear subject. A listener must be trained to know how to take the test properly, and have opportunity to become familiar with the set up, the content, what differences may be, etc. We do not want them to be dumb and stupid machines and just push buttons. We want to extract knowledge so anything we can do to guide a test subject to find the differences that are there, must be done.

Being an expert/golden ear listener is very different. That is a process of going through extensive training across a ton of tests, learning the algorithms in question, participating in many tests, interacting with other experts, formal training, etc. It took me some 6 months to finish this process and come out the other end as a trained listener.

It is like you learning to play a few note on a Piano from an expert. That one session does not make you a Piano player and certainly no expert.

There are basic, basic things about how to do the tests properly. But they are foreign to people online who think the only qualification to take a test is to not be deaf as FMW wrongly said.

Don't know what this stream of consciousness means. The test set out to disprove the position you all hold and it did so in spades. It won an award for best peer reviewed conference paper. No Joe random forum poster gets to reposition it as something else, when even the vocabulary of conducting double blind tests is not understood.

"Laughable?" Unbelievable. Cry should be the operative word if I were in your shoes...

If it is exhausting, you may want to refrain from manufacturing things. It certainly is exhausting for me to keep responding to incorrect things you are saying in these posts. Here is yet another example in the form of Ethan's test that I said I ran: https://www.avsforum.com/forum/91-aud...l#post25869354

Just look a few posts up and you see me responding to your post. So you read about this other test.

Here is me in that very same thread responding to you implying I can't pass a double blind test of 320 kbps MP3 against the original: https://www.avsforum.com/forum/91-aud...l#post26197034

All on AVS. All in front of you.

You attempted to belittle the results of these positive outcomes and I am here to correct you. Stick with the facts and real data and I won't post. It is that simple.

You will never know given your test methods. Anyway, I don't believe you are interested in answering this.

Posted by Desertdome in the kc meet thread. The kc guys have done a number of blind and DBT tests

That's nicely done. The pacing is perfect.

Nope, they're all still here, reading and learning. ...When they'll have smart questions, they'll know exactly where to ask them and to who.

In that the context of the cited article is based on speaker evaluation, a few things come to mind:


1. All speaker tests, are cited evaluations. There is no retail outlet, that offers properly controlled DB ABX tests of any kind, let alone speakers. Making DB ABX comparisons moot.


2. The Sonic differences between speakers, has been well evidenced as being easily discernable, by even amateur observers, visa vie DB ABX test performed in years gone by.


3. Sight evaluations of CD Transports, DAC, and Amplifiers and Preamps, is a different story, as most DB ABX tests have produced null's as outcomes; however, Sighted evaluations have produced weighted outcomes...


4. Sighted evaluations which were performed with the equipment cover up, have produced similar results to that of DB ABX tests (not including speakers).


5. Sighted evaluations are arguably, the only metric available to the masses, and I would argue relentlessly that 99% of AVS members and onlookers within this forum, have made their buying decision visa vie sighted evaluations, and that furthermore, that the loins share is completely satisfied with their decisions(s)!


6. BD ABX testing is extremely complex to proctor, on many levels... I don't know of a single QSC comparator in use and available to even a smallish portion of the population. Only conjecture can exist in the current landscape. But again, empirically, sighted evaluations have resulted in hundreds of millions of satisfied customers.


7. I don't believe that their bad, poor or any such thing - they're certainly not dishonest! Removing sighted evaluations, and leaving only the verbal regaling's of sales personnel to depict the performance attributes of products, would absolutely be a dishonest evaluation metric. After all, by what means would the sale staff garner their opinions? Please don't say technical specifications. And we all know that it wouldn't be visa vie BD ABX testing's...


8. Sighted evaluations are simply all that's within a scope of practical implementation and therefore reach. Without them, we would truly be making a blind decision.

I love your eight points above, in particular the last one.

In the link contained in the quoting post above (original first post), on Sean's blog Alan Sircom, pro audio reviewer from the UK - Hi-Fi+ -, and also a member over @ WBF (and most likely here too @ AVSForum), occasionally posting intelligently, had a very realistic comment for Dr. Sean Olive; it's worth double-checking, and Sean's reply.

____________


Nice to see new faces around with well spoken facts.

I agree with all of your points except for #7 . Normally, a sighted evaluation will produce exactly the results that the salesperson suggested so that makes it dishonest in my book. I think there might be some value in sighted evaluations in rare cases but those cases would normally involve people who have no expectations. I can't imagine that it would involve anybody reading this thread and that is our audience. The salesman's comments and the sighted evaluation are the same thing in most cases.


What is a consumer to do? Choose what sounds best in whatever way it is possible to compare. But the informed consumer will do that with some knowledge of what blind tests have shown. As an example, I would never buy an external DAC for any reason other than for unusual switching and connection capabilities because I've been through the tests and I know better.

No, it's not "thrown in your face", you have plenty of time to listen to the sound, etc.

I have no idea what you're even imagining here, it sounds like some gross corruption of how any reasonable test should be run.