AVS Forum banner

1 - 20 of 261 Posts

·
Registered
Joined
·
16,166 Posts
Discussion Starter #1 (Edited)
This is a splinter thread from the Marantz AV7005 thread.

Duckstu asks:
Quote: Originally Posted by duckstu
What if one unit A sounds better (to the person doing the test) at both higher volume, similar volume, and lower volume levels than unit B?

Is the opinion of the person doing the test still invalid?

And should that person be barred from mentioning their opinion of the two units?
When scientists typically conduct A/B or A/B/X tests of high fidelity audio products in a blind or better still double blind, controlled environment, they are not asking the test subject's opinion as to a preference or "Which is better, in your opinion?". That's subjective and has no real answer. The only question on the table is "Can you tell any difference between between A vs B?" or in the case of ABX testing "Which is this unknown unit called X? Is it this one you are welcome to listen to as much as you'd like called A, or this other one you are welcome to listen to as much as you'd like called B?"

In these tests there is no "opinion" or "personal taste" involved. The only thing being determined is whether the small difference between the two DUTs (devices under test) are sufficiently different such that a human can perceive the difference, or not.

In such tests it is important to play the two DUTs at exactly the same volume, aka level [within a .1 dB tolerance, usually], as measured by external test instruments electrically, which can detect differences of level even smaller than the human ear can, so that the the test subjects aren't getting a tell tale clue as to the identity of each DUT when it gets played. [On a side note, studies show humans typically falsely attribute small changes in elevated level, say a half dB or so, as an improvement in "quality", oddly, and not what it really is, quantity, but again that has nothing to do with an A/B test which doesn't concern itself with "quality" or "personal preference" but instead only if there is a discernible difference or not.

Here's an analogy. If A/B testing two colas (soda pop), for example, if one of the two sodas was served at an ever so slightly colder temperature than the other, then the test subjects may, often at a subconscious level, notice that difference (and not the actual inherent flavor of the soda), yet they may not be aware that the difference they are keying on is the temperature and not the taste.
 

·
Registered
Joined
·
18,923 Posts
I think few people would debate the procedure you outline, assuming they find merit with controlled blind listening tests (not everyone agrees with it.)
 

·
Banned
Joined
·
18,829 Posts

Quote:
Originally Posted by m. zillch /forum/post/20395299


When scientists typically conduct A/B or A/B/X tests of audio products in a blind or better still double blind, controlled environment, they are not asking the test subject's opinion as to a preference or "Which is better, in your opinion?". That's subjective and has no real answer. The only question on the table is "Can you tell any difference between between A vs B?" or in the case of ABX testing "Which is this unknown unit called X. Is it this one you are welcome to listen to as much as you'd like called A, or this other one you are welcome to listen to as much as you'd like called B?"

That is only one type of ABX blind test. The other type is to ask for a score of say, 1 to 5 relative to the original. The averaged results is called MOS or Mean Opinion Score.

Quote:
In these tests there is no "opinion" or "personal taste" involved.

Clearly the "opinion" is involved because that is what the tester is indicating. Personal taste also comes into play and strongly so when we test large impairments. For example, in testing two video codecs, a subject may like a softer image better than a sharper one with more artifacts. Whereas another tester may be quite the opposite.

Quote:
In such tests it is important to play the two DUTs at exactly the same volume, aka level [within a .1 dB tolerance, usually], as measured by external test instruments electrically, which can detect differences of level even smaller than the human ear can, so that the the test subjects aren't getting a tell tale clue as to the identity of each DUT when it gets played.

This is more of an ideal than something that can be fully achieved. Think how you can for example match the levels of two speakers. As you know, speakers have lots of variation in their frequency response. Matching them becomes quite problematic. Good number papers have come out on various ways this can be managed to some extent but of course, cannot match the ideal.


If the responses are flat then yes, level matching can be done. We can also take advantage of hearing threshold and relax the requirement to some extent at both extremes of frequency response.


Going to OP's test, he is not running an ABX. He is running an AB test. There is no "reference" in his model. He has two pieces of equipment and is making a subjective call as to which one is better sounding. That type of testing is far more realistic for enthusiasts than true ABX which requires computer tracking and gadgets to simulate ABX testing.


In the sense of taking a shortcut then, I would tend to agree with his thesis that if a piece of equipment sounds better at any volume, then it tends to be better. And that strict level matching while compromised is followed in spirit. Lots of people will debate that and indeed, have done that to me
. Regardless of debate, I would give lots of opportunity to anyone who has taken the time to run such a test to share it with us. He certainly stands head and shoulders above people who advocate perfect tests, but never run them themselves, nor would know how to create the fixture to do so.
 

·
Banned
Joined
·
18,829 Posts

Quote:
Originally Posted by MichaelJHuman /forum/post/20395469


I think few people would debate the procedure you outline, assuming they find merit with controlled blind listening tests (not everyone agrees with it.)

Oops
.
 

·
Registered
Joined
·
16,166 Posts
Discussion Starter #5
Amirm, you probably haven't read the 5000+ posts in the other thread that lead to this thread, as I have, but our ongoing discussion there was re. amps and preamps, both of which have a relatively flat frequency response, unlike speakers, (and therefore are quite a bit easier to level match) and if they are audibly different, or not.


The introduction of topics like video codecs, quality scoring, speakers and other devices with irregular, non-flat frequency responses, etc. is not part of our discussion.


People in that thread were upset this general amp difference audibility discussion was off topic and they wanted to only discuss the specific Marantz AV7005 unit, so I started this thread.


You are welcome to stay, but please limit your discussion to only amps, preamps, and level matching them (or not) in blind testing.
 

·
Banned
Joined
·
18,829 Posts

Quote:
Originally Posted by m. zillch /forum/post/20395676


The introduction of topics like video codecs, quality scoring, speakers and other devices with irregular, non-flat frequency responses, etc. is not part of our discussion.

While some do, with the advent of class D amps, you can't make that assumption anymore. Here is the frequency response of NAD M2 amplifier:



You will level match this how? Which load would you use and why?

Quote:
You are welcome to stay, but please limit your discussion to only amps, preamps, and level matching them in blind testing.

I guess thanks are in order for letting me stay
.


Keep in mind that many of the points I made apply just the same. You are not likely able to set up ABX tests at home. Practically wins over perfection if you care to have more data than just words to argue about.
 

·
Registered
Joined
·
18,923 Posts
No matter how you slice it, any thread of this sort will go wildly off topic. Never seen it not happen. I am sure your intentions are noble, but you should know the drill by now. The usual suspects will show up, and declare their positions, and due to the (sometimes amusing) imperfections of language and communication being imperfect, people will start making analogies to Corvettes vs Yugos, and it will all go downhill



p.s. I would be very uncomfortable buying a class D amp who's graph looked like that. I assume though, that perfectly reasonable class D amps are made, and more will be made, that avoid that sort of issue. But I agree 100%. It makes no sense to assume all amps have ruler flat response when some don't. But most do, I think?
 

·
Registered
Joined
·
16,166 Posts
Discussion Starter #8 (Edited)
Amirm, your graph is meaningless to us without labeling the 4 color traces [or a source link would probably suffice .]
 

·
Registered
Joined
·
1,805 Posts
Quote:
Originally Posted by guitaraficionado
As you may have learned, any post asserting a listener's preference for a component not under "ideal laboratory" conditions warrants a flippant response from m. zillc and WiWavelength demanding a level-matched scientific analysis of the experience.
When your testing method is the equivalent of licking your thumb, sticking it into the wind, and telling us that the high temperature today will be higher than it was yesterday, expect a flippant response. Your informal experience with a component tells us very little, if anything, reliable about the comparative fidelity of the device under test. In fact, such can be highly misleading to both yourself & others.


As for your preference, that is nothing but idle chit chat, about as useful as posting "I really like chocolate ice cream."


AJ
 

·
Registered
Joined
·
6,946 Posts

Quote:
Originally Posted by WiWavelength /forum/post/20403606


As for your preference, that is nothing but idle chit chat, about as useful as posting "I really like chocolate ice cream."


AJ

So what?


There is no requirement regarding the real or perceived value of a post/thread.
 

·
Registered
Joined
·
1,805 Posts

Quote:
Originally Posted by whoaru99 /forum/post/20404443


So what?

Yes, exactly, you unknowingly support my point. We should all be free to question opinions, especially when they arise from dubious origins.

Quote:
Originally Posted by whoaru99 /forum/post/20404443


There is no requirement regarding the real or perceived value of a post/thread.

Agreed. And there is no prohibition on posts critiquing the methodology behind and/or validity of other posted assertions.


AJ
 

·
The Village Idiot
Joined
·
9,625 Posts
Interesting. Since sound is in the end subjective when utilizing the human ear to evaluate it - and there is no getting around that - it's the proverbial beating of a dead horse.


I guess I'll get some popcorn and watch/read what happens in this thread. My prediction - it'll end up like all the rest. Not everyone is going to agree with the results garnered from subjective tests. Especially if/when their favorite speaker/amp/receiver/preamp etc gets thrashed in the finals.


In other words - we are witness to yet another pissing contest...
 

·
Registered
Joined
·
8,956 Posts
As always, there are two fundamental questions here: Can people distinguish between sources in a properly designed blind test? And, how can a test be properly designed?
 

·
Registered
Joined
·
2,470 Posts

Quote:
Originally Posted by amirm /forum/post/20395582


That is only one type of ABX blind test. The other type is to ask for a score of say, 1 to 5 relative to the original. The averaged results is called MOS or Mean Opinion Score.

In an ABX test, you are asked to determine whether "X" is A or B, correct? How does a scale of 1 to 5 do that?
 

·
Registered
Joined
·
18,923 Posts
I don't trust subjective opinions much...for many, many reasons.


Been reading about audio since I was a teen, and I am no spring chicken. I got really tired of certain review styles that were loaded with ambiguous terms for one thing. How do I know the reviewer can really swap in some piece of gear, change nothing else and then figure out how good some gear is? I have my doubts about many reviewers being able to remove all biases.


Your average consumer reporting on their gear is even more questionable due to their exuberance and pride of ownership. I tend to be ultra-critical of my new gear and would rarely say I hear improvements in the sound. But some people are very different, understandably so.


Then you have the ubiquitous night and day comments here..."I changed cables, and heard a night and day difference."


Then you have people saying there's differences in things that no logic or scientific stretch can explain such as a power cable making a difference in sound.


Then you have people stating polar opposite positions on products. "Wow the Denon RX-1000 is amazing" "Oh yeah, I think the Denon RX-1000 sucks, so much worse than my NAD T-10,000"


Then you have people not controlling conditions making their conclusions highly suspect. "Well, I changed out everything, and the soundstage opened up, must have been the receiver."


Then you have the questionable ability of people's audio memory to actually remember what they heard.


All that being said, I think a trained listener can listen for certain things. For example, I was able to figure out what to listen to when comparing MP3 files...cymbals (after going back and forth between the same song with different encodings). Only real difference I heard between mid rate and high rate MP3 files was in the high end. I am sure there are many examples of people learning to listen to very specific things in a sample piece of music, and hear differences. (I am not saying I am a trained listener, it was an example of what can be learned, I think



But if you just tell people to determine which of two amps sound better, and the conditions are not tightly controlled, and they are not trained listeners who have some reference (such as listening to the sample audio many times through neutral headphones or something,) than what good is the test?


Just my opinion.
 

·
Registered
Joined
·
319 Posts
As has been mentioned, it's hard to level-match well, and listening over long periods with small relative differences plus or minus, should be adequate to make an initial conclusion.


What hasn't been mentioned is the vast differences in hearing, that alter our perceptions. Beethoven had incredible audio memory, history reports he could recall a previously unheard concert note by note verbatim. One wonders how much detail he retained beyond the timing and pitch.
 

·
The Village Idiot
Joined
·
9,625 Posts

Quote:
Originally Posted by guitaraficionado /forum/post/20406206


As has been mentioned, it's hard to level-match well, and listening over long periods with small relative differences plus or minus, should be adequate to make an initial conclusion.


What hasn't been mentioned is the vast differences in hearing, that alter our perceptions. Beethoven had incredible audio memory, history reports he could recall a previously unheard concert note by note verbatim. One wonders how much detail he retained beyond the timing and pitch.

Your signature says it all - thats all that matters to me in the end.

FWIW when there is general consensus on what sounds good I most often agree - it usually sounds good to me too. Not very often have I wondered what the hell the reviewer might have heard that I missed. I remember when I first read a glowing review (1970 or so) of the much vaunted - often lampooned Bose 901s - a friend and I went down to take a listen to see what the hubbub was all about. My friend was awed by them - they were LOUD as hell - I kept asking the sales guy what Bose did with the cymbals and other high notes - and the lows that were as clear as the Mississippi at flood stage. He ignored me. I was able to talk my friend out of buying them since they were priced higher than a new motorcycle at that time. He ended up with a pair of RS Mach 1s. Great sounding speakers for the times - with highs and lows and everything in between.
 

·
Registered
Joined
·
1,805 Posts

Quote:
Originally Posted by guitaraficionado /forum/post/20406206


As has been mentioned, it's hard to level-match well, and listening over long periods with small relative differences plus or minus, should be adequate to make an initial conclusion.

Perform the test unsighted. If you consistently pick the same component, regardless of level higher or lower than that of the other component under test, then, yes, you may be able to "make an initial conclusion." On the other hand, if you insist on performing the test sighted, then all bets are off. Too many variables are in play. Your mind may predispose you, for example, to sighted component "B" because of prior knowledge: name, price, aesthetics, circuit topology, etc. And then your mind convinces itself that "B" at a lower level does not sound just softer but "smoother, more delicate," while "A" at a higher level does not sound just louder but "harsh & fatiguing."

Quote:
Originally Posted by guitaraficionado /forum/post/20406206


What hasn't been mentioned is the vast differences in hearing, that alter our perceptions. Beethoven had incredible audio memory, history reports he could recall a previously unheard concert note by note verbatim. One wonders how much detail he retained beyond the timing and pitch.

Some people are able to run a sub 4 minute mile, some to memorize pi to 10,000 decimal places, and some, probably, to distinguish between 0.04% THD and 0.08% THD. But odds are that you (collectively speaking) are not one of those people. Audiophilia does not bestow upon you superhuman hearing. If you actually have supremely acute auditory perception, then you should be able demonstrate that by passing an ABX test.


AJ
 
1 - 20 of 261 Posts
Top