AVS Forum banner
Status
Not open for further replies.

Debate Thread: Scott's Hi-res Audio Test

132K views 3K replies 74 participants last post by  markrubin 
#1 ·
As a courtesy to Scott Wilkinson, I thought any debate about hi-res vs CD should be here. As such, I am taking the posts from that thread and answering them here:
Quote:
Originally Posted by antoniobiz1  /t/1531590/avs-aix-high-res-audio-test-proposed-methodology/30#post_24726545


As stated in page 776 _ "Many types of music and voice signals were included in the sources, from classical (choral, chamber, piano, orchestral) to jazz, pop and rock music."


So, either such a wide sample means that no sacd or dvd-a had high res content, or Meyer and Moran were really really really...unlucky
.


Turns out luck is not a factor but proper protocol is. If you were taking a drug and it turned out that in 20% of the cases they mixed up the placebo for the real drug in the tests, would you feel comfortable continuing to take that drug? I know I would not. You might say, what if that wouldn't have changed the outcome? My answer would be still the same. There are some mistakes you just don't make. In the case of a drug trial it would be keeping the samples straight. That is why should such protocol errors surface, the entire study gets thrown out. You sort of can't believe anything else they have done as being accurate.


By the same token, if you are going to test "high-res" audio vs 44.1 Khz, the first and foremost thing to do is to make sure you are bloody testing high-res content. If you didn't think or know to perform a simple spectrum comparison, well, I am not going to put faith in anything else you are reporting. Mind you, I think their results are probably right but I simply can't wear the cloth of objectivity and then turn around and look the other way when folks make beginner mistakes.


Some people have no business running such tests. If you can't be super careful and check and re-check everything three times, especially if you are going to write a paper to be published in a Journal of the industry, then you have no business running such tests. BTW, the "problem" becomes ours if we start to parade this in public as proper evidence. We would be tainted by their mistakes. I know I don't want to go there. I hope others do not either.


So sadly we have to throw this "evidence" away. There is no such thing as being part-time vegetarian.
Either a test is reliable and authoritative or it is not. There is no in between.


If still not convinced, think how you would behave if the very test had found a difference. Wouldn't' we try to dismiss their results by saying, "look here, you had two files that were the same and you still found them to be different?" If the same mistake is cause for dismissal of one outcome, objectivity calls that it does the same in reverse. We have to play by our rules. We can't play by other people's rule where we are trying to invalidate their beliefs.
 
See less See more
2
#2 ·
One would think that with all the criticism of the Meyer Moran tests, there would be people EAGER to repeat the identical conditions with "real" hires (funny how people were gushing over how much better and closer to analog the "faux hires" recordings were than CD) recordings to show them how wrong they were, but they haven't. It's rather obvious why...


Quote from a different forum on this topic:
Quote:
there were enough genuine hi-res recordings in that list (at least, recordings with real high frequency content above 22kHz) to allow people to ABX those if they were in any way special / sonically above what CD can achieve.

And this:
Quote:
Fans are/were forever wetting themselves about how these older recordings sound so much better on hi-res than CD.


The idea that 15ips analogue recordings don't sound better on hi-res than CD is only mentioned to try to debunk Mayer and Moran. Outside this specific purpose, fans of hi-res seem convinced that


Indeed! This Patricia Barber, Nightclub is well regarded for its really impressive quality.

You will find nearly on every review a remark about the better sounding SACD layer even if there is not anything above 22kHz because the original recording was done on an old digital machine. So if MOFI didn´t mess up the CD-Layer this one should have been a pro HiRes one even without extended HF content. I bet so it goes for other releases listed also.

So i don´t think the reasoning of BW limited sources count at all.
Quote:
think how you would behave if the very test had found a difference.
Behave how? If a difference was found, it's logical to assume that "true" hires material would have shown an even greater difference.
 
#3 ·
Anybody familiar with this study?
http://www.aes.org/e-lib/browse.cfm?elib=15398
Quote:
It is currently common practice for sound engineers to record digital music using high-resolution formats, and then down sample the files to 44.1kHz for commercial release. This study aims at investigating whether listeners can perceive differences between musical files recorded at 44.1kHz and 88.2kHz with the same analog chain and type of AD-converter. Sixteen expert listeners were asked to compare 3 versions (44.1kHz, 88.2kHz and the 88.2kHz version down-sampled to 44.1kHz) of 5 musical excerpts in a blind ABX task. Overall, participants were able to discriminate between files recorded at 88.2kHz and their 44.1kHz down-sampled version. Furthermore, for the orchestral excerpt, they were able to discriminate between files recorded at 88.2kHz and files recorded at 44.1kHz.


Authors: Pras, Amandine; Guastavino, Catherine

Affiliation: McGill University, Montreal, Quebec, Canada
 
#6 ·

Quote:
Originally Posted by amirm  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24728070


As a courtesy to Scott Wilkinson, I thought any debate about hi-res vs CD should be here. As such, I am taking the posts from that thread and answering them here:
.


Turns out luck is not a factor but proper protocol is. If you were taking a drug and it turned out that in 20% of the cases they mixed up the placebo for the real drug in the tests, would you feel comfortable continuing to take that drug? I know I would not. You might say, what if that wouldn't have changed the outcome? My answer would be still the same. There are some mistakes you just don't make. In the case of a drug trial it would be keeping the samples straight. That is why should such protocol errors surface, the entire study gets thrown out. You sort of can't believe anything else they have done as being accurate.

The tested drugs, in Meyer and Moran, are SACD and DVD Audio, as they existed at the time. No custom recordings involved. Just commercially available discs, released to the public and purchasable in most record stores.

The placebo was CD audio Redbook quality (downsampling to 16bit/44.1).


There was no drug mixing whatsoever. At worst, you can say that, at that time, they were not able to fully exploit SACD and DVD/Audio formats.


As per the quoted list, 19 discs were tested, out of a 5000 titles available. 19 is a pretty big sample (can anyone help me with statistic significance?) Again, at worst you can say that in general, existing recordings DO NOT benefit from the hires treatment.


Which leads to another interesting point, by the way: if the "great flaw" of Meyer and Moran is lack of hypersonic content, that means that no recording lacking hypersonic content will benefit from hires treatment.


Which in turn means that anything recorderd prior to 1980 will not need hires treatment (forget Beatles, Eagles and Elvis and Jaco and Duke Ellington). No recording from digital multitrack of the eighties and the nineties (Sony DASH, etc.) will benefit either (except those recorded at 20 bit, which are not a lot, in my experience). That's a LOT of music.
 
#2,807 ·

The tested drugs, in Meyer and Moran, are SACD and DVD Audio, as they existed at the time. No custom recordings involved. Just commercially available discs, released to the public and purchasable in most record stores.
The placebo was CD audio Redbook quality (downsampling to 16bit/44.1).
1) The necessary protocols are nothing like a drug test.

2) The three probe signals were CD, SACD, and DVD.

3) CD is not "placebo", it's just one of the test conditions.

Calling CD a "placebo" means you're absolutely sure it does nothing. I'm not even going to go into that here. Or anywhere, really.
 
  • Like
Reactions: Garidy
#7 ·
If the content that was being tested was not hi-res and one was of the opinion that this leaves the conclusions drawn to be specious, then wouldn't a reasonable course of action be to repeat the study using such material? While this could be done by the original authors, others could certainly investigate it, no?
 
  • Like
Reactions: John Schuermann
#8 ·

Quote:
Originally Posted by Roseval  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24728687


Anybody familiar with this study?
http://www.aes.org/e-lib/browse.cfm?elib=15398
I am and have the paper also. Will post relevant results.
Quote:
Originally Posted by RobertR  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24728703


Discussed here.
There is nothing but conjecture in that post. It does show what I said: when the shoe is on the other foot, all of a sudden we become highly critical. With Meyer and Moran test we have real, objective, verifiable issues with their testing. The post you linked to from Arny is just shooting in the dark hoping it hits something
.


Here is a sample comment: "There have been many improper jobs of resampling that have lead to misleading audible differences. However we have long (over a decade) had good resampling software such as Cool Edit Pro and Audition."


I don't know what improper jobs Arny is talking about. Would be good to see a list of those tests. But it matters not here: either he knows the test used "improper resampling" or he doesn't. If he doesn't have such data, he is creating FUD . That is not objective.


Interesting that he talks about using Cool Edit Pro and Audition. That is the tool that would have found the problems with Meyer and Moran tests.


His comments at least in that post indicates to me that he has not read the paper itself. I have not however read the rest of that lengthy thread. If you have, do you mind linking to Arny actually saying that and quoting anything other than the abstract that is available to all?
 
#9 ·

Quote:
Originally Posted by Chu Gai  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24729118


If the content that was being tested was not hi-res and one was of the opinion that this leaves the conclusions drawn to be specious, then wouldn't a reasonable course of action be to repeat the study using such material? While this could be done by the original authors, others could certainly investigate it, no?
Sure. But the implication that nobody has is not data that we can use to draw conclusions from. That would be a "trust me" argument. What you are really implying is "trust me, if the flaws were removed the conclusions would be the same." Problem is, the people we want to convince don't trust us. They see us repeatedly posting this test without putting in a strong disclaimer that issues were found with it. And that no professional testing effort would have made such a mistake. Lack of transparency on our part causes us to have no credibility. So "trust me' arguments on test results we don't have doesn't work.


It is a shame really as the trust me argument should have some value. Hopefully we practice more transparency in the future and start to gain such credibility.
 
#10 ·
No, I'm not making a trust me argument, Amir. I'm simply saying this would be fertile ground to explore and i would hope someone would.
 
#11 ·

Quote:
Originally Posted by antoniobiz1  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24728765


The tested drugs, in Meyer and Moran, are SACD and DVD Audio, as they existed at the time. No custom recordings involved. Just commercially available discs, released to the public and purchasable in most record stores.

The placebo was CD audio Redbook quality (downsampling to 16bit/44.1).


There was no drug mixing whatsoever.
Of course there were. They were testing the idea of whether frequency response above 22.05 Khz of 44.1 Khz sampling rate matters or not. They presented two samples to the user: one that was supposed to have content above that (i.e. the high-res sample) and the other not (the 44.1 Khz version). Problem is, the former was not high-res in all cases. They were presenting the same samples in a way to the user. The equiv. thing in drug testing would be to have given the real drug to both camps where one was supposed to get the get placebo.


Fact that they were a victim of commercial recordings being that way is an excuse, not an explanation that leads us to trusting the data and experimenters. It just boggles the mind that they would not test the before and after files to see if they were different objectively. They just stuck a label on a track as being "high--res" because they *thought* it was. That is not how we perform proper testing. We don't rely on thought. We rely on data.
Quote:
At worst, you can say that, at that time, they were not able to fully exploit SACD and DVD/Audio formats.
No, at worst we can say that the rest of the report is faulty just as well. We simply can't trust the experimenters to have done the rest of the test right. We have data that we can gather on the high-frequency response. We can't know if they made other operational errors because we were not there and don't have a recording of the whole process. It is unfortunate that we get to that conclusion as I appreciate the work they did. But we must go there given the fundamental error.
Quote:
As per the quoted list, 19 discs were tested, out of a 5000 titles available. 19 is a pretty big sample (can anyone help me with statistic significance?) Again, at worst you can say that in general, existing recordings DO NOT benefit from the hires treatment.
We can't say that about quality of commercial discs. Indeed the report confirms that:

Though our tests failed to substantiate the claimed advantages of high-resolution encoding for two-channel audio, one trend became obvious very quickly and held up throughout our testing: virtually all of the SACD and DVD-A recordings sounded better than most CDs—sometimes much better. Had we not “degraded” the sound to CD quality and blind-tested for audible differences, we would have been tempted to ascribe this sonic superiority to the recording processes used to make them.


Plausible reasons for the remarkable sound quality of these recordings emerged in discussions with some of the engineers currently working on such projects. This portion of the business is a niche market in which the end users are preselected, both for their aural acuity and for their willingness to buy expensive equipment, set it up correctly, and listen carefully in a low-noise environment.


Partly because these recordings have not captured a large portion of the consumer market for music, engineers and producers are being given the freedom to produce recordings that sound as good as they can make them, without having to compress or equalize the signal to suit lesser systems and casual listening conditions. These recordings seem to have been made with great care and manifest affection, by engineers trying to please themselves and their peers. They sound like it, label after label. High-resolution audio discs do not have the overwhelming majority of the program material crammed into the top 20 (or even 10) dB of the available dynamic range, as so many CDs today do."



It matters not at the end why the discs sounded better. They did. And that is what we desire as consumers: better sound. Fact that without having ultrasonics preserved they still accomplished the same is neither here, nor there.
Quote:
Which leads to another interesting point, by the way: if the "great flaw" of Meyer and Moran is lack of hypersonic content, that means that no recording lacking hypersonic content will benefit from hires treatment.


Which in turn means that anything recorderd prior to 1980 will not need hires treatment (forget Beatles, Eagles and Elvis and Jaco and Duke Ellington). No recording from digital multitrack of the eighties and the nineties (Sony DASH, etc.) will benefit either (except those recorded at 20 bit, which are not a lot, in my experience). That's a LOT of music.
I hear what you are saying
. But we don't have such data. A single test and sampling done by people who are not familiar with basics of how you do such tests properly cannot be the basis for further conclusions. We would need to cite other work that is not from the same source.


Personally I don't care about this argument itself. What I like to see is a digital file as was produced in the studio. If it were an old analog tape or whatever and was captured at 96 Khz, I *want that file*. I don't want them to resample it for me to 44.1 Khz. I can do that as can anyone else here that can spend two minutes learning how to use a resampling program. Disk storage is not a concern anymore and neither is bandwidth. The whole reasoning why we have a problem with high-res is a stale argument looking for a problem.


Why would we advocate that the file be converted to a lower resolution before we get it? I mean it is not like we have an argument that doing so makes it better. I hope we are in violent agreement that at best, it will be equal to the master. In no case would it be better. If so, then let's advocate getting the original file. If you must have the 44.1 Khz version, then the distributors will offer that also. The world has come to peace with this already but we keep pretending the argument has a valid use. It doesn't. Sites like HD Tracks are providing high-res files and they are not going to close their doors because we wave a four year old flawed study in their face and that of their customers.
 
#12 ·

Quote:
Originally Posted by Chu Gai  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24729870


No, I'm not making a trust me argument, Amir. I'm simply saying this would be fertile ground to explore and i would hope someone would.
Ok, my apology for assuming so
. Then I agree that there should have been more tests. I have one other that I will try to find and post.
 
#13 ·
Apology accepted! How's the smoker doing?
 
#15 ·
The M&M test used sacd and with dsd there is *always* >20Khz content: DSD Noise (and lots of it).


Some if the tested disc had 20 or 24 bit sources (48kHz) and analog tape sources converted to 24 bit digital It may be only (tape) noise content in stead of music signals but it still counts as content that should contribute to be able to detect an audible difference.


Above all the M&M test proved that inserting a adc and dac is transparent.
 
#16 ·

Quote:
Originally Posted by amirm  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24729849


the people we want to convince don't trust us.
Many people who believe in the superiority of hires wouldn't trust any data using any methodology that doesn't conform with what they think they already "know". You know perfectly well that this is the case.
Quote:
It matters not at the end why the discs sounded better
Of course it matters. People would know that careful production of 16/44 gives them just as good sound as hires.
 
#17 ·

Quote:
Originally Posted by RobertR  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730131


Many people who believe in the superiority of hires wouldn't trust any data using any methodology that doesn't conform with what they think they already "know". You know perfectly well that this is the case.
So what is the motivation for constanting debating this topic if you believe the other side will not trust a word you are saying?
 
#18 ·

Quote:
Originally Posted by amirm  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730141


So what is the motivation for constanting debating this topic if you believe the other side will not trust a word you are saying?
You missed where I said "many" and not "all". Convincing the likes of audiophools like Robert Hartley and John Atkinson is not the goal. Enlightening others who aren't of such a mindset is.
 
#19 ·

Quote:
Originally Posted by RobertR  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730151


... Convincing the likes of audiophools like Robert Hartley and John Atkinson is not the goal. Enlightening others who aren't of such a mindset is.
Yes, they will never accept the facts and will carry their beliefs to the grave.
And yes, that is for sure about the latter.
 
#20 ·

Quote:
Originally Posted by amirm  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24729825


.... With Meyer and Moran test we have real, objective, verifiable issues with their testing. ...

Well, you don't if what is said that a number of those recordings indeed contained ultrasonic material.

And, you just cannot compare the Conference paper which is not peer reviewed to one that was.

Yes, the authors could repeat it but why bother? Something else will not be acceptable.

How about you doing such an endeavor?
 
#21 ·

Quote:
Originally Posted by amirm  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730141


So what is the motivation for constanting debating this topic if you believe the other side will not trust a word you are saying?
Simple. One cannot walk away from pseudoscience and snake oil claims unchallenged or, these will become the accepted facts even if they are false.

Perhaps our ancestors should have overlooked the witch burnings and just accepted that there were witches? And, don't tell me this is not a proper analogy.
 
#22 ·

Quote:
Originally Posted by CharlesJ  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730255


Yes, the authors could repeat it but why bother? Something else will not be acceptable.
Anything that doesn't conclude that hires is audibly superior to CD is unacceptable to some people.
Quote:
How about you doing such an endeavor?
Since the sole goal seems to be to discredit M&M and not find out the audibility of hires vs. CD, I don't think he's interested.
 
#23 ·

Quote:
Originally Posted by CharlesJ  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730255


Well, you don't if what is said that a number of those recordings indeed contained ultrasonic material.
Please read my earlier reasoning as to why we can't trust much of that report given the basic failures in creating the test. As I said, maybe those results are correct despite lack of proper protocol. Convention says we should not put such weight behind them. It is unfortunate that we have to dismiss the totality of their work but that is required of "us" if we are to live by our rules of proper and careful testing protocols.
Quote:
And, you just cannot compare the Conference paper which is not peer reviewed to one that was.
I have always reminded folks to not put too much weight behind "peer review" and this is a great example of it. When I was at Microsoft my signal processing PhD's would be the peer reviewers for other conferences. I asked them what they did and they said they would look to see if the person flunked school and such. In no way are they assuring that the conclusions in the paper are right or wrong. They can't do that. What they can do is find a person saying 1+1 = 3 and rejecting the paper on the basis of the person not being to do simple math (in my trivial example here).


Given this, I perform the same analysis myself. I look at the work and see if they made simple, simple mistakes. Here is an example. A proper double blind study will have a control. The control is something where we know for sure the answer. If our test then shows that people are voting otherwise, either the test is wrong or the subjects are. If the testers here knew about this simple protocol, they would have been forced to analyze the spectrum of the files and discover what we know now, i.e. their test samples did not represent what they set out to test.
Quote:
Yes, the authors could repeat it but why bother? Something else will not be acceptable.
That's true. The other camp will always find something wrong including the whole DBT affair not being acceptable to them. But all is not lost. People like me would then defend their work if they did not make the most basic mistakes. I most believe in their outcome yet I can't stand next to them. I just can't. No way would I ever create a test that would have such obvious failings. And it is not just this one test. I will later post other tests from the same group that had multiplicity of errors.
Quote:
How about you doing such an endeavor?
You are expecting me to be less lazy than I am.
Blind tests are the most boring things to set up and run. I paid my dues when it was part of my job and that of my group to perform them. That is an important thing by the way: my career and company's success depended on whether we had constructed correct tests or not. Hobbyist such as the group that M&M are part of don't have that requirement. Sure there is some egg on their face due to these errors but their next house mortgage payment does not depend on it. If it did, I think they would be much less likely to make such errors. They would have read a lot more tests of these types, performed many others, be familiar with others doing the same kind of testing, etc., etc. Just because you are a hobbyist listening to music and got an ABX box doesn't make you the proper person to run such tests as we see here. There is real science here that is deeper than the casual treatment we often give it.


Back to me, my current project is to get my CNC router up and running. Need to make a lot of things with it that are far more interesting than dealing with some philosophical matter about high-res audio which has no real life consequence. As I said, folks are free to download high-res files today and there is no longer a format war. You want only 44.1 Khz stuff, you can buy the CD or download the same. You can even go lower to MP3 and AAC. There is nothing to be accomplished one way or the other. If others feel strongly about it and are qualified to properly create such a test, I would be happy to see the results of their work.
 
#24 ·

Quote:
Originally Posted by RobertR  /t/1532092/debate-thread-scotts-hi-res-audio-test#post_24730151


You missed where I said "many" and not "all". Convincing the likes of audiophools like Robert Hartley and John Atkinson is not the goal. Enlightening others who aren't of such a mindset is.
My bad. I thought when you said "many people" you meant more than a couple of people. Addressing that regardless, I am confident there is nothing we can do to challenge their views of audio especially since they are not even here to have a discussion with them. Even if they were, what good comes out of that? If you are a devout Christian and some dude goes around elsewhere and shouts there is no god, would you be up in arms about it? I hope not
.


The reason I am engaging in the discussion is that I hope that we reform ourselves. We are here. We seem to have lots of interest in the topic. So there is opportunity for learning. Learning what constitutes a good test. Learning to be neutral with our criticism of tests even if they favor our point of view. Learning how the technology really works. And value of measurements and psychoacoustics science in that regard. We seem to want to charge ahead without any of this. That can’t be right if we are going to be vocal about this topic.


Speaking of measurements, I am and I hope many of you are also, in debt of John Atkinson for the incredible library of free measurement data about audio gear. I don’t know how one can call himself an “objectivist” yet find no value in that gift that stereophile has provided to us. No subscription fee, no nothing. I often see Arny for example using their measurements to make his point. So not sure I buy into him being of the top two villains.


I personally ignore everything RH says and subjective things that JA may say (outside of anything related to a measurement). Problem solved.
 
Status
Not open for further replies.
You have insufficient privileges to reply here.
Top