or Connect
AVS › AVS Forum › Audio › Receivers, Amps, and Processors › The Subjective and Objective Evaluation of Room Correction Products
New Posts  All Forums:Forum Nav:

The Subjective and Objective Evaluation of Room Correction Products - Page 10

post #271 of 582
Quote:
Originally Posted by Kal Rubinson View Post

AudysseyPro lets you see the measured response and lets you control the target curve (within limits).

Are these frequency response curves any different than what the SMS-1 would show after Audyssey is run?
post #272 of 582
Quote:
Originally Posted by Kal Rubinson View Post

AudysseyPro lets you see the measured response and lets you control the target curve (within limits).

I thought Audyssey Pro didn't actually measure the final room response, but created a graph showing what it computed the final response would be after applying Audyssey correction.
post #273 of 582
Quote:
Originally Posted by Kal Rubinson View Post

AudysseyPro lets you see the measured response ......

Aren't they a bit "optomistic"? And didn't you do secondary measurements on both ARC & Audyssey in one of your reviews?
post #274 of 582
Thread Starter 
Quote:
Originally Posted by pepar View Post

Fair enough. Are eight trained listeners and three songs each played three times is sufficient to do that?

Someone asked me previously, "What is psychoacoustics?" I define it as the relationship between the perception and measurement of sound. This experiment was designed to have listeners evaluate the sound quality of different room corrections that were then correlated with some objective measurements. The results indicate good correlation between the different subjective ratings (preference, spectral balance and related comments) and in-room measurements of the room corrections.

The results of our test indicate that using 8 trained listeners, 3 programs with 3 repeats provided sufficient statistical confidence with these variables to make some conclusions.Moreover, the fact that the subjective ratings can be explained by objective measurements, and are in general agreement with current psychoacoustic knowledge about perception of loudspeakers and rooms suggests that the listening test results are not by chance alone.

Yes, it would be nice to use more trained listeners, and more programs but I don't think it would fundamentally change the conclusions. As I've stated in the conclusion section of the paper, the quality and quantity of bass in a recording will always be a nuisance factor in listening tests. We also need to repeat the study in more listening rooms using more loudspeakers in order to improve the degree to which the results can be generalized to conditions outside the ones tested here. This study is just the first part of hopefully many studies on room correction to follow.

Do the results apply to untrained listeners? Based on prior experience, I would likely have to use 100-200 untrained listeners to get the same statistical reliability as using trained listeners. In a previous study on loudspeaker preferences, I found both trained and untrained listeners essentially preferred the same loudspeakers, and these preferences could be largely explained by the loudspeaker frequency response. I would expect the same general rules for trained versus untrained listeners to hold true for room correction, but this needs to be experimentally confirmed.
post #275 of 582
Thread Starter 
Quote:
Originally Posted by Roger Dressler View Post

Thanks much for the detail.

Do you put any stock in Chris K's contention that even though one might create an identical frequency response correction filter using either PEQ, IIR or FIR, only FIR need apply due to optimal phase properties?

If these various implementations do in fact result in different phase yet similar freq response, might not there be some resultant audible difference?

I've also seen multiband octave equalizer that allows independent adjustment of level and phase. As if it matters.

I realize that none of this proves anything. It just raises the question.

I have seen no experimental evidence in the scientific literature to indicate that FIR filters have any psychoacoustic benefit over IIR filters in their application to loudspeakers and room correction. My personal view, which is shared by many respected audio scientists including Stanley Lipshitz and Floyd Toole, is that FIR filters only add cost and unnecessary complexity to loudspeaker/room correction, with no apparent gain in performance. They also give marketing and sales departments something to talk about (hence the confusion that you suffer from ).
post #276 of 582
Thread Starter 
Quote:
Originally Posted by THE_COW_IS_OK View Post

Just want to chime in and thank Tom for all the info he disclose. The dynamics behind his efforts are irrelevant to me as long as I look at the data/results objectively. I do enjoy reading his articles in his blog along with the free linked AES papers! I place Harman articles on bass reproduction along with Tool book on the same rank with Everest books in terms of educational merit.

Now back to the topic

Considering the ear/brain ability to compensate for tonal imbalances when given enough, time, I am interested in knowing if listeners we capable of switching among different RCs at will or some kind of waiting period was mandatory. The latter is prefered since the first would give a preference advantage to FR with mean performance among the FR RC candidates.

Hi THE_COW_IS_OK:

Thanks for your compliments and comments. BTW, there is a new edition of Alton Everest's book available that has been co-authored by Ken Pohlmann. If you read it you will find some discrepancies between Everest's and Toole's view of small room acoustics.

I think you are talking about loudspeaker/room adaptation. In our experiments, listeners were able to switch immediately back and forth at will between the different room corrections, thus minimizing opportunities to adapt to the various room corrections. This multiple comparison method is a well-accepted method (see ITU-R BS 775 ?) to evaluate loudspeakers, amplifiers or medium-low quality perceptual CODECS because it provides the most discriminating and reliable responses. I don't agree that we should have allowed listeners to adapt to the different room corrections since the purpose of the study was to measure audible differences in room correction -- rather than measure adaptation to room correction.

My research suggests adaptation to room acoustics happens very quickly - within seconds of hearing different loudspeakers in different rooms. We've also found that within the context of comparing different virtual automotive audio systems, listeners adapt quickly to certain non-indivdualized calibration errors in the auditory display. After a few seconds, the auditory display errors that remain constant while switching among different virtual cars, tend to be ignored.
post #277 of 582
So I am curious, are people thinking Audyssey is RC3 and Anthem is RC5?
post #278 of 582
I was assuming Audyssey was RC6 because of the 2Khz dip.
post #279 of 582
Quote:
Originally Posted by millerwill View Post

Are these frequency response curves any different than what the SMS-1 would show after Audyssey is run?

Sure. They are measured differently.

Quote:
Originally Posted by Steve Dodds View Post

I thought Audyssey Pro didn't actually measure the final room response, but created a graph showing what it computed the final response would be after applying Audyssey correction.

I was talking about the initial measured response prior to correction.

Quote:
Originally Posted by rnrgagne View Post

Aren't they a bit "optomistic"?

See above.

Quote:


And didn't you do secondary measurements on both ARC & Audyssey in one of your reviews?

Yes.
Sigh.
post #280 of 582
Quote:
Originally Posted by Kal Rubinson View Post

Sure. They are measured differently.

Can you explain? I know that Aud uses time as well as freq info when setting its filters, but when Aud shows a spectrum (FR)--i.e., intensity versus frequency--I fail to see how this could be any different from the spectrum another instrument would measure (unless, of course, one instrument is simply in error). ???
post #281 of 582
Quote:
Originally Posted by millerwill View Post

Can you explain? I know that Aud uses time as well as freq info when setting its filters, but when Aud shows a spectrum (FR)--i.e., intensity versus frequency--I fail to see how this could be any different from the spectrum another instrument would measure (unless, of course, one instrument is simply in error). ???

Audyssey's measurements are based on multiple mic positions.
post #282 of 582
Quote:
Originally Posted by Kal Rubinson View Post

Audyssey's measurements are based on multiple mic positions.

Ah, thanks Kal; that I do understand!

I have a SMS-1 that I use after Audyssey (yes, I know, many recommend it the other way around), and I can certainly see how the FR changes when I hold the mike in different locations, e.g., a foot or two to the left, right, forward, back of the prime location (very much the locations I use when running Audyssey). It's very illuminating to see how much (or how little) the FR changes with position; the SMS-1 is very helpful and simple to use in this regard.
post #283 of 582
Quote:
Originally Posted by Tonmeister2008 View Post

The results of our test indicate that using 8 trained listeners, 3 programs with 3 repeats provided sufficient statistical confidence with these variables to make some conclusions.Moreover, the fact that the subjective ratings can be explained by objective measurements, and are in general agreement with current psychoacoustic knowledge about perception of loudspeakers and rooms suggests that the listening test results are not by chance alone.

Yes, it would be nice to use more trained listeners, and more programs but I don't think it would fundamentally change the conclusions. As I've stated in the conclusion section of the paper, the quality and quantity of bass in a recording will always be a nuisance factor in listening tests. We also need to repeat the study in more listening rooms using more loudspeakers in order to improve the degree to which the results can be generalized to conditions outside the ones tested here. This study is just the first part of hopefully many studies on room correction to follow.

Do the results apply to untrained listeners? Based on prior experience, I would likely have to use 100-200 untrained listeners to get the same statistical reliability as using trained listeners. In a previous study on loudspeaker preferences, I found both trained and untrained listeners essentially preferred the same loudspeakers, and these preferences could be largely explained by the loudspeaker frequency response. I would expect the same general rules for trained versus untrained listeners to hold true for room correction, but this needs to be experimentally confirmed.

Thank you. I look forward to tests with a larger number of listeners.

What are your thoughts on the issue of comparing room correction technologies whose goal is to improve listening at multiple seats with technologies designed to improve listening at one seat?

Jeff
post #284 of 582
Quote:
Originally Posted by dweltman View Post

So I am curious, are people thinking Audyssey is RC3 and Anthem is RC5?

Guesses from a friend who keeps up with these sorts of things (and is free to reveal himself if he wishes):

RC1,2: Harman
RC3: Anthem (room gain bump)
RC4: nothing
RC5: Audyssey
RC6: Lyngdorf (just because it has anecdotally the worst performance)

5 and 6 could be changed around, because of the 2kHz dip.

edit: Just as important as the final results are, it's also the ease by which a user can get to those final results that are important. Sean mentioned some as being especially difficult to use, so the final rankings may be affected if you can't get an RC to perform at its full potential.

--Andre
post #285 of 582
Quote:
Originally Posted by Tonmeister2008 View Post

As I've stated in the conclusion section of the paper, the quality and quantity of bass in a recording will always be a nuisance factor in listening tests.

Why is the nuisance factor limited to the bass quality in recordings? Isn't there equal chance that the treble or the midrange quality will vary enough between recordings that it could sway the results?
post #286 of 582
Quote:
Originally Posted by dweltman View Post

So I am curious, are people thinking Audyssey is RC3 and Anthem is RC5?

It would be unlikely that Audyssey is RC3, unless they modified the target curve during the test to have a raise in the LF. By default, Audyssey corrects to flat LF at the measurement position.
post #287 of 582
Quote:
Originally Posted by hifisponge View Post

By default, Audyssey corrects to flat LF at the measurement position.

Which, for better or worse, is perceived as thin bass. Hence the preference in these tests for room gain at the lower frequencies.

Presentation Slides

Looking at slides 23 and 24, only two room corrections seem to be correcting to flat in the low frequencies: RC5 and RC6. So I'm guessing one of them is Audyssey, probably RC5.

If you look at the corresponding Spectral Balance plot of RC5 on slide 25, you'll notice that the listeners thought it was dropping as the frequencies got lower, when that's NOT what was happening according to the measurements. Reality vs perception (human hearing).

No wonder thin bass was an early complaint about Audyssey. Next go-round should re-test with Audyssey's Dynamic EQ (though I don't know if that feature is available in their external box).

By comparison, look at the spectral balance for RC1 and RC2. Both are tilted noticably downward as the frequencies get higher, but both were perceived as pretty flat.

If I were a manufacturer of a room correction system, I would be paying attention to these tests, especially when it comes to what listeners prefer. Harman is footing the bill, but everyone gets the benefits of the resulting data.
post #288 of 582
Quote:
Originally Posted by sdurani View Post

If I were a manufacturer of a room correction system, I would be paying attention to these tests, especially when it comes to what listeners prefer. Harman is footing the bill, but everyone gets the benefits of the resulting data.

Isn't it likely that any company with a room correction product has already done their own tests and trust their data more?
post #289 of 582
But then we come again to the question whether we should aim for reference or preference.

Given Audyssey's position on this, surely they would discount listening tests (preference) in favour of measurements (reference)?
post #290 of 582
Quote:
Originally Posted by millerwill View Post

Can you explain? I know that Aud uses time as well as freq info when setting its filters, but when Aud shows a spectrum (FR)--i.e., intensity versus frequency--I fail to see how this could be any different from the spectrum another instrument would measure (unless, of course, one instrument is simply in error). ???

Quote:
Originally Posted by Kal Rubinson View Post

Audyssey's measurements are based on multiple mic positions.

Hi,

As Kal points out, the Audyssey frequency response curves consider multiple measurements, but unlike other approaches, they don't spatially average the results. They use fuzzy logic to weigh the measurements based on common characteristics. So while the curves may be close, they will not be identical to spatially averaged measurements.

Larry
post #291 of 582
Quote:
Originally Posted by pepar View Post

Isn't it likely that any company with a room correction product has already done their own tests and trust their data more?

Sure, but this is the first I've heard of anyone testing what listeners prefer, let alone trying to correlate measurements with those preferences. Most manufacturers of room correction products start with an a priori goal of flat response (reference), irrespective of whether consumers do or don't like the resulting sound (preference). So it wouln't hurt to consider some of the data from this test, especially since it's free and not likely being done by other manufacturers of room correction.
post #292 of 582
Quote:
Originally Posted by millerwill View Post

Ah, thanks Kal; that I do understand!

I have a SMS-1 that I use after Audyssey (yes, I know, many recommend it the other way around), and I can certainly see how the FR changes when I hold the mike in different locations, e.g., a foot or two to the left, right, forward, back of the prime location (very much the locations I use when running Audyssey). It's very illuminating to see how much (or how little) the FR changes with position; the SMS-1 is very helpful and simple to use in this regard.

Hi,

To use the SMS-1 and come closer to what Audyssey does (or what Sean did in his test) you can purchase Velodyne's MIC-5 add-on. This is an array of five microphones that attach to the SMS-1 and instead of showing the results at a single location, up to five locations can be measured simultaneously and shown on the SMS-1 display. It is averaging the responses.

Larry
post #293 of 582
Quote:
Originally Posted by Steve Dodds View Post

But then we come again to the question whether we should aim for reference or preference.

Given Audyssey's position on this, surely they would discount listening tests (preference) in favour of measurements (reference)?

From what I've read, their technology is based on extensive listening tests that were - and are - conducted at USC. That Tom Holman is rooted in the film industry and has been an evangelist for standards suggests to me that reference is their goal. And many, many posts from their CTO confirm that.

Jeff
post #294 of 582
Quote:
Originally Posted by sdurani View Post

Sure, but this is the first I've heard of anyone testing what listeners prefer, let alone trying to correlate measurements with those preferences. Most manufacturers of room correction products start with an a priori goal of flat response (reference), irrespective of whether consumers do or don't like the resulting sound (preference). So it wouln't hurt to consider some of the data from this test, especially since it's free and not likely being done by other manufacturers of room correction.

Those are good points, but I doubt that Audyssey is going to change their philosophy. And where does educating the consumer come in? I think that quite enough companies over the decades have tailored their speaker products to the curve du jour. Goosed bottom end, sweetened midrange, etc - I'd liken it to Pepsi being able to pull Coca Cola off their game with the Pepsi Challenge and the natural human preference for the sweeter of two drinks.

So, IMO, that's what companies are doing when they give much of the weight to preferences- loading up on the sugar. Trained listeners? Wonder what the training was?

Jeff
post #295 of 582
Quote:
Originally Posted by pepar View Post

So, IMO, that's what companies are doing when they give much of the weight to preferences- loading up on the sugar. Trained listeners? Wonder what the training was?

Trained listeners have been the cornerstone of Sean's work (and indeed any other good studies on perception) for years now. His blog has more info.

The first link talks about what he trains his listeners for. The second about how trained listeners perform much better than untrained listeners, and the third about how trained and untrained listeners have the same preferences.

http://seanolive.blogspot.com/2009/0...-listener.html
http://seanolive.blogspot.com/2008/1...mances-of.html
http://seanolive.blogspot.com/2008/1...f-trained.html

--Andre
post #296 of 582
Quote:
Originally Posted by sdurani View Post

Which, for better or worse, is perceived as thin bass. Hence the preference in these tests for room gain at the lower frequencies.

Presentation Slides

Looking at slides 23 and 24, only two room corrections seem to be correcting to flat in the low frequencies: RC5 and RC6. So I'm guessing one of them is Audyssey, probably RC5.

If you look at the corresponding Spectral Balance plot of RC5 on slide 25, you'll notice that the listeners thought it was dropping as the frequencies got lower, when that's NOT what was happening according to the measurements. Reality vs perception (human hearing).

No wonder thin bass was an early complaint about Audyssey. Next go-round should re-test with Audyssey's Dynamic EQ (though I don't know if that feature is available in their external box).

By comparison, look at the spectral balance for RC1 and RC2. Both are tilted noticably downward as the frequencies get higher, but both were perceived as pretty flat.

If I were a manufacturer of a room correction system, I would be paying attention to these tests, especially when it comes to what listeners prefer. Harman is footing the bill, but everyone gets the benefits of the resulting data.

You're preaching to the choir.

I use manual PEQ and my target is and has been +6dB @ 30Hz relative to the midrange (typcial "house curve"). Though, I like a bit more sparkle to the top-end, so the response from 1KHz to 10kHz is EQ'd to be flat at the LP (which would be a rising treble response nearfield).

Here's a mind bender. If the music recording industry embraced a standard that used the shelved down FR response found in RC1 as their reference, then would we need to target flat response in the playback system?
post #297 of 582
Quote:
Originally Posted by AndreYew View Post

Trained listeners have been the cornerstone of Sean's work (and indeed any other good studies on perception) for years now. His blog has more info.

The first link talks about what he trains his listeners for. The second about how trained listeners perform much better than untrained listeners, and the third about how trained and untrained listeners have the same preferences.

http://seanolive.blogspot.com/2009/0...-listener.html
http://seanolive.blogspot.com/2008/1...mances-of.html
http://seanolive.blogspot.com/2008/1...f-trained.html

--Andre

Thanks! That's going to take me a while to digest.

Jeff
post #298 of 582
Quote:
Originally Posted by hifisponge View Post

Here's a mind bender. If the music recording industry embraced a standard that used the shelved down FR response found in RC1 as their reference, then would we need to target flat response in the playback system?

The frequency responses Sean posted is the In Room response. The room reinforces the lower frequencies while dampening the higher ones. If we removed the effects of the room Toole, Harman et al., have shown the resulting Anechoic response (on axis) is very flat and smooth, with the typical sloping in level as we move off axis. This has been Harman's position for a long time. So really the requirement of flat and smooth -anechoically- has not changed. Now how much slope there is to the off axis responses is important (and also contributes to the in room response having a downward slope), and thankfully Sean has published the slope (for a specified measurement condition) of the best 10% of loudspeakers reviewed by Harman in an AES paper number 6190. Well worth the $20 if you are not a member.
post #299 of 582
The thing is, the tilted curve doesn't really show a non-flat speaker and it's not some sort of fashionable curve du jour. If you put a speaker that's flat on axis in a good room and measure a 'steady-state response' (pink noise, stepped sine, ungated impulse, etc) it will show a tilted-down response. Trying to keep the steady-state in-room response flat will give you a tilted-up on-axis response which just doesn't sound good. It's all a matter of how you measure and the target curve has to adjust to the measurement method.

Edit: Brandon beat me to it.
post #300 of 582
BTW if one were willing to use a computer in their playback chain there is no reason one couldn't use the EQ software posted by Dennis (catapult) and use this target slope shown by Harman to be the most desirable. I believe Dennis has already plotted it and it could be converted to text file for use by the EQ software easily. For everyone else you'll just have to buy the standalone RC devices.

Keep in mind I don't believe this will correct the sins of the loudspeaker design in the first place, but it would be better than nothing. As Sean's test shows.
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Receivers, Amps, and Processors
AVS › AVS Forum › Audio › Receivers, Amps, and Processors › The Subjective and Objective Evaluation of Room Correction Products