AVS › AVS Forum › Audio › Receivers, Amps, and Processors › How do I verify - or debunk - the claims of The Upgrade Company?
New Posts  All Forums:Forum Nav:

How do I verify - or debunk - the claims of The Upgrade Company? - Page 17

post #481 of 1596
4/1/12 at 4:23pm
silly question to ask, did you do a complete reset of the system settings prior to doing the blind tests... ?????
Reply
Reply

AVS Top Picks

post #482 of 1596
4/1/12 at 4:31pm
Jeff, thanks for holding course and getting this done. Kudos to you and to everyone involved.

Quote:
Originally Posted by pepar View Post

Overall - any of the others that want to step up and post their individual results that is up to them - the group preferred the modded unit 17 out of 33 times - 51.51% of the time. As a group overall, we correctly identified X only 42% of the time (14/33). All of us did better on the stereo test than the MCH test.

I do have one conclusion based on this test and that is the The Upgrade Company fails miserably in meeting the lofty claims on their website. And their going on and on about "For Profit Built" competitors is a huge helping of barnyard dung ... as if they aren't in it for profit. Sheesh.

Should be interesting to see what Mr. S and his disciples have to say about this. I'm gonna guess they'll blame it on the switch.
Reply
Reply
post #483 of 1596
4/1/12 at 4:57pm
Thread Starter 
Quote:
Originally Posted by craig john View Post

Jeff,

The night before the test, when we set up Dennis' 885 with your 5508, we noticed there were some level differences between the 2 units, especially in the MC section. IIRC, we had a 4.5 to 5.5 dB offset between them when we played SACD's and DVD-A's. Did you notice any of those kinds of differences between the levels of the modded and un-modded 5508's? Did you need to make any offsets of level settings between the 2 units, or were the levels, trims and MCV always the same?'

Craig

Housekeeping - the speakers are Legacy Audio Signature SE (3) and Legacy Audio Phantom HD surrounds

Amp: Enlightened Audio Designs (EAD) PowerMaster 5300 (5x300W)

IN comparing the 885 and the 5508 we noticed a big difference in the way they each handled SACD with - for SACD only - a four or five dB volume control difference needing to be used to balance them. We did not notice a similar difference in the way they handled any other source, nor did we notice and differences in the way the two 5508s handled anything.

Two more things - get Dennis to tell you about his angry phone call from ....

And Dennis is going to ask the switch loaner if he can keep it a while longer as we would like you and the gang to do a blind A/B/X test with the 885 and the 5508, for our own curiosity. Dennis is .. struggling a bit on considering that he might be susceptible to suggestion as to the sound quality differences between the stock and modded 5508s.
Reply
Reply
post #484 of 1596
4/1/12 at 4:58pm
Thread Starter 
Quote:
Originally Posted by RUR View Post

Jeff, thanks for holding course and getting this done. Kudos to you and to everyone involved.

Should be interesting to see what Mr. S and his disciples have to say about this. I'm gonna guess they'll blame it on the switch.

He already has.
Reply
Reply
post #485 of 1596
4/1/12 at 4:59pm
Thread Starter 
Quote:
Originally Posted by mystic_sniper28 View Post

silly question to ask, did you do a complete reset of the system settings prior to doing the blind tests... ?????

What?
Reply
Reply
post #486 of 1596
4/1/12 at 5:01pm
Quote:
Originally Posted by pepar View Post

And Dennis is going to ask the switch loaner if he can keep it a while longer as we would like you and the gang to do a blind A/B/X test with the 885 and the 5508, for our own curiosity. Dennis is .. struggling a bit on considering that he might be susceptible to suggestion as to the sound quality differences between the stock and modded 5508s.

Since I am interested in these results, you may certainly keep the switch for a bit longer.

BTW, from the data, despite some seeming preference for the modified unit, it seems that none of the results are significantly different from chance.
Reply
Reply
post #487 of 1596
4/1/12 at 5:02pm
Quote:
Originally Posted by mystic_sniper28 View Post

silly question to ask, did you do a complete reset of the system settings prior to doing the blind tests... ?????

Yes, we set both prepros exactly the same. Audyssey was turned off for the test. For the stereo listening, we used Pure Audio mode which bypasses both the video circuits and the bass management system - we were only listening to the L and R speakers, full range (stereo). The speakers produce down to 20 Hz measured in my room. For multichannel listening, we did not use any processing modes, just the native MC format like "DSD" for SACDs, and "Multichannel" for DVD-As.

BTW, the system included 3 matching Legacy Audio Signature SEs for the L/C/R channels, and a pair of Legacy Audio Phantom HDs for the surrounds.

Amplification was an Enlightened Audio Design (EAD) PowerMaster 5300 amp (5x300W). An Oppo BDP-83SE was used as the source.
Reply
Reply
post #488 of 1596
4/1/12 at 5:05pm
Quote:
Originally Posted by craig john View Post

Jeff,

The night before the test, when we set up Dennis' 885 with your 5508, we noticed there were some level differences between the 2 units, especially in the MC section. IIRC, we had a 4.5 to 5.5 dB offset between them when we played SACD's and DVD-A's. Did you notice any of those kinds of differences between the levels of the modded and un-modded 5508's? Did you need to make any offsets of level settings between the 2 units, or were the levels, trims and MCV always the same?'

Craig

Craig - we checked for this and since both units were the same (Onkyo PR-SC5508), the volume controls tracked identically on CDs, SACDs, and DVD-As, once both unit's internal levels were level-matched using an SPL meter.
Reply
Reply
post #489 of 1596
4/1/12 at 5:05pm
Thread Starter 
Quote:
Originally Posted by Kal Rubinson View Post

Since I am interested in these results, you may certainly keep the switch for a bit longer.

BTW, from the data, despite some seeming preference for the modified unit, it seems that none of the results are significantly different from chance.

Yes, that is true. Though overall I preferred the stock unit 73% of the time. And that is a huge head-scratcher for me.
Reply
Reply
post #490 of 1596
4/1/12 at 5:07pm
Quote:
Originally Posted by pepar View Post

On the stereo test, my results were 5/6 (83%) correctly identified as "X" and - ta da! - 5/6 (83%) I preferred the STOCK unit. On the 5.1, I missed them all bit preferred the stock unit 3/5 (60%). Over the two tests, I preferred the stock unit 8/11 (73%). I am flummoxed as I choose the one I thought to have the better inner detail, something the sighted portion had me thinking modded.

Overall - any of the others that want to step up and post their individual results that is up to them - the group preferred the modded unit 17 out of 33 times - 51.51% of the time. As a group overall, we correctly identified X only 42% of the time (14/33). All of us did better on the stereo test than the MCH test.

Quote:
Originally Posted by Kal Rubinson View Post

BTW, from the data, despite some seeming preference for the modified unit, it seems that none of the results are significantly different from chance.

Agreed, Kal. If the units could not be correctly identified, then "preference" is meaningless. If one can't tell one unit from the other, one can't *have* a preference.

Craig
Reply
Reply
post #491 of 1596
4/1/12 at 5:13pm
Quote:
Originally Posted by craig john View Post

If the units could not be correctly identified, then "preference" is meaningless. If one can't tell one unit from the other, one can't *have* a preference.

We should chisel your statement in granite.

BTW, that's why I urged that they test for preference in addition to the ABX.
Reply
Reply
post #492 of 1596
4/1/12 at 5:18pm
Quote:
Originally Posted by Kal Rubinson View Post

Since I am interested in these results, you may certainly keep the switch for a bit longer.

BTW, from the data, despite some seeming preference for the modified unit, it seems that none of the results are significantly different from chance.

Kal, thanks alot. We will also include a pink noise track in subsequent tests as another data point.

What would you say about a test that only used pink noise for every trial? Would removing musical progam material from an ABX test and just using test tones make the results more or less accurate? Does using music in the trials add too many variables to the test, if all you are trying to prove is if you can pick out one unit from another blind?

I found it difficult to use music tracks where things I was listening for maybe were 30 or 40 seconds into the track... Then, rewinding to the beginning, and waiting another 30-40 seconds, then rewinding again. That made it very hard to compare A, B, and X in my mind with such seperation in between...

However, when we did instantaneous switching between A and B, we all agreed we could hear differences. Why did an ABX test not be able to show the same results? I would be very curious to see if using a continuous signal - like a pink noise test tone - would change the results of the test, since it may be easier to compare the tonality difference between DUTs using pink noise...

Any thoughts?
Reply
Reply
post #493 of 1596
4/1/12 at 5:26pm
Thread Starter 
Quote:
Originally Posted by DMark1 View Post


However, when we did instantaneous switching between A and B, we all agreed we could hear differences.

It was sighted. 100% of the time in the sighted listening, I preferred the modded unit.

In fact, we all did!
Reply
Reply
post #494 of 1596
4/1/12 at 5:28pm
Quote:
Originally Posted by DMark1 View Post

Kal, thanks alot. We will also include a pink noise track in subsequent tests as another data point.

What would you say about a test that only used pink noise for every trial? Would removing musical progam material from an ABX test and just using test tones make the results more or less accurate? Does using music in the trials add too many variables to the test, if all you are trying to prove is if you can pick out one unit from another blind?

I found it difficult to use music tracks where things I was listening for maybe were 30 or 40 seconds into the track... Then, rewinding to the beginning, and waiting another 30-40 seconds, then rewinding again. That made it very hard to compare A, B, and X in my mind with such seperation in between...

However, when we did instantaneous switching between A and B, we all agreed we could hear differences. Why did an ABX test not be able to show the same results? I would be very curious to see if using a continuous signal - like a pink noise test tone - would change the results of the test, since it may be easier to compare the tonality difference between DUTs using pink noise...

Any thoughts?

If you can consistently identify one unit's pink noise from the other unit's pink noise, then you can have a preference for one unit's pink noise over the other unit's pink noise. But would you????

I'm just kidding Dennis. Actually pink noise could be a more reliable way to ensure you can identify one unit from the other. However, it can't help with preference. For that, you need music or some other content.

Craig
Reply
Reply
post #495 of 1596
4/1/12 at 5:28pm
Thread Starter 
Quote:
Originally Posted by pepar View Post


I dare - no double dog dare - The Upgrade Company to submit one of their modded units for a proper double-blind test conducted by a neutral third party. It'll never happen.

Jeff

I am going to retract this. I don't trust that we wouldn't be getting a ringer.
Reply
Reply
post #496 of 1596
4/1/12 at 5:31pm
Thread Starter 
Quote:
Originally Posted by DMark1 View Post


However, when we did instantaneous switching between A and B, we all agreed we could hear differences. Why did an ABX test not be able to show the same results? I would be very curious to see if using a continuous signal - like a pink noise test tone - would change the results of the test, since it may be easier to compare the tonality difference between DUTs using pink noise...

Any thoughts?

I have to wonder if any frequency response differences wouldn't have been EQ'd out if we had set up Audyssey? If mods boost the presence range, which we surmised was the case with the pink noise differences we heard, would that "survive" running room correction?
Reply
Reply
post #497 of 1596
4/1/12 at 5:37pm
Quote:
Originally Posted by pepar View Post

I have to wonder if any frequency response differences wouldn't have been EQ'd out if we had set up Audyssey? If mods boost the presence range, which we surmised was the case with the pink noise differences we heard, would that "survive" running room correction?

A very good point Jeff. I like your idea about a single point Audyssey calibration for both DUTs.. that would be another very interesting test.
Reply
Reply
post #498 of 1596
4/1/12 at 5:38pm
Any word from the 3rd participant?
Reply
Reply
post #499 of 1596
4/1/12 at 5:43pm
Thread Starter 
Quote:
Originally Posted by craig john View Post

Any word from the 3rd participant?

None that I know of. Dennis?

I'm wondering when we will hear from ... the elephant in the room?
Reply
Reply
post #500 of 1596
4/1/12 at 5:43pm
Quote:
Originally Posted by pepar View Post

It was sighted. 100% of the time in the sighted listening, I preferred the modded unit.

In fact, we all did!

That's what I am trying to reconcile in my brain!! If the stock unit sounded better to me sighted, I would've said so. But it didn't! And we all agreed! So why such a vastly different result in a blind ABX test??? I'm trying to understand this and stay objective, as a scientist would...
Reply
Reply
post #501 of 1596
4/1/12 at 5:43pm
Quote:
Originally Posted by craig john View Post

If you can consistently identify one unit's pink noise from the other unit's pink noise, then you can have a preference for one unit's pink noise over the other unit's pink noise. But would you????

Doing ABX with pink noise eliminates more variables but some of those variables might be significant. Transient response, for one, might be an inconsequential parameter in pink noise which, although random, is subjectively continuous. Still, pink noise ABX would provide some sort of base line reference.

Quote:
I'm just kidding Dennis. Actually pink noise could be a more reliable way to ensure you can identify one unit from the other. However, it can't help with preference. For that, you need music or some other content.

I prefer some pinks over other pinks.
Reply
Reply
post #502 of 1596
4/1/12 at 5:46pm
Quote:
Originally Posted by DMark1 View Post

A very good point Jeff. I like your idea about a single point Audyssey calibration for both DUTs.. that would be another very interesting test.

That is doable only if the EQ is identical. That requires AudysseyPro privileges for both units with one doing the measurements and having AudysseyPro transfer those measurements to the other. Not trivial.
Reply
Reply
post #503 of 1596
4/1/12 at 5:47pm
Thread Starter 
Quote:
Originally Posted by DMark1 View Post

That's what I am trying to reconcile in my brain!! If the stock unit sounded better to me sighted, I would've said so. But it didn't! And we all agreed! So why such a vastly different result in a blind ABX test??? I'm trying to understand this and stay objective, as a scientist would...

The mind is a terrible thing ...
Reply
Reply
post #504 of 1596
4/1/12 at 5:48pm
Quote:
Originally Posted by pepar View Post

None that I know of. Dennis?

I spoke with him on the phone. He believes we did a fair test, although he is disappointed with his "performance" on the test. Of course, he was expecting a different outcome, clearly identifying the modded unit with statistical certainty. He does not seem to blame the switch, (although DS did).
Reply
Reply
post #505 of 1596
4/1/12 at 5:50pm
Thread Starter 
Quote:
Originally Posted by Kal Rubinson View Post

That is doable only if the EQ is identical. That requires AudysseyPro privileges for both units with one doing the measurements and having AudysseyPro transfer those measurements to the other. Not trivial.

MultEQ Pro requires minimum of three measurements, which requires moving the mic. And that would introduce a variable. A single measurement, the min for MultEQ XT, would involve the mic on a stand and untouched between calibrations.

Wouldn't that result in identical calibrations?
Reply
Reply
post #506 of 1596
4/1/12 at 5:51pm
Thread Starter 
Quote:
Originally Posted by Kal Rubinson View Post


I prefer some pinks over other pinks.

Or a rosé instead.
Reply
Reply
post #507 of 1596
4/1/12 at 5:56pm
Thread Starter 
Quote:
Originally Posted by DMark1 View Post

I spoke with him on the phone. He believes we did a fair test, although he is disappointed with his "performance" on the test. Of course, he was expecting a different outcome, clearly identifying the modded unit with statistical certainty. He does not seem to blame the switch, (although DS did).

I'm glad you heard from him and that he thinks it was fair. I thought it was as fair as it could be short of having it take place in academia.

We should invite him for the stock 885 vs stock 5508 test.

Jeff
Reply
Reply
post #508 of 1596
4/1/12 at 5:57pm
Quote:
Originally Posted by pepar View Post

MultEQ Pro requires minimum of three measurements, which requires moving the mic. And that would introduce a variable. A single measurement, the min for MultEQ XT, would involve the mic on a stand and untouched between calibrations.

Wouldn't that result in identical calibrations?

XT32 on my 80.2, (identical to 5508), requires 3 measurements before it will calculate.

Craig
Reply
Reply
post #509 of 1596
4/1/12 at 6:00pm
Thread Starter 
Quote:
Originally Posted by craig john View Post

XT32 on my 80.2, (identical to 5508), requires 3 measurements before it will calculate.

Craig

OK, so much for that idea, though we could have three stands to reduce/eliminate that variable.
Reply
Reply
post #510 of 1596
4/1/12 at 6:03pm
Quote:
Originally Posted by craig john View Post

XT32 on my 80.2, (identical to 5508), requires 3 measurements before it will calculate.

Craig

Does the SW allow you to use the same point in space, or is it too smart for that old trick?
Reply
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Receivers, Amps, and Processors

AVS Top Picks

AVS › AVS Forum › Audio › Receivers, Amps, and Processors › How do I verify - or debunk - the claims of The Upgrade Company?