Originally Posted by DMark1
Very interesting thoughts Craig... I know ABX is commonly regarded as the gold standard in audio tests, but you may have a point that the ABX test may not be a "high sensitivity" test - which may account for results that say two DUTs sound the same, when there really is a measurable difference between them.
In that case, all ABX blind tests conducted to date are flawed. That seems most unlikely to me.
I think the biggest flaw in the test conducted wrt to this specific thread is that no objective measurements were taken of the two units under test. If measuring equipment had been available it would have been a straightforward matter to test unit A and unit B and to see first of all if there is any objective difference between input and output of both units. If there is no difference, then the two units will perform identically. If there is a difference, then the question becomes 'is the difference audible?' and finally the question is 'which of the two different presentations do I prefer?'.
The listening tests can then be compared with the measurements and a check made to see how the measurements corroborate, or not, the audible differences. For example, if the measurements show an extended HF response in one unit, and that one unit is always observed as being "more open and transparent, more airy, in the treble" then one can see that the listening test results are grounded in scientific fact. If OTOH, nobody can hear any difference which would corroborate the extended HF in one unit, then one can come to the conclusion that the modifications did indeed make a difference (extended HF) but the difference had no audible effect.
I am not knocking the test conducted and I know how difficult it is to cover all the bases, especially when someone suggests using very expensive test gear which may not be available to the testers.
The problem is always going to be that the aficionados of the modifications will always point to flaws in the test if the result does not favour their modified unit. If, of course, the results did favour their modified unit, then the test would be 'perfect' and would 'prove the point'. Such is the nature of subjectivity and subjectivists.
My own personal take on this requires no further tests at all: the customer is promised a hugely significant upgrade in performance, such that his modified unit will equal or better competitive units costing $10,000 or even $20,000. That there is so much discussion as to whether any difference in SQ at all
exists, indicates to me that the promise of the manufacturer has not been met. If he had achieved his goal and fulfilled the promises on his website, the audible differences would be heard by all. If they do exist they are simply too small to warrant the price asked and the equivalent money would bring greater returns if spent elsewhere, on proven modifications that can be easily demonstrated to bring significant improvements to SQ - eg bass traps.