Originally Posted by MoltenLava
DBT is a test procedure that's very good at showing whether A and B are different. Those advocating DBT on this forum failed to realize is DBT is not a sufficient procedure to determine A and B are the same. When A and B are sufficiently similar, then it takes well chosen samples to show that A and B are indeed different. It's much easier to come up with a set of samples, either by mistake or intentionally, to show A and B are the same. But that does not prove that A and B are the same, as DBT is not designed to prove such.
Totally agree, and really it needs another experiment type to probably back up the results.
I would be curious to see the results where there is just product A and product B and one song.
The listener is given the chance to become familiar with product A listening to 1 song several times.
Now just before the test starts, product A or product B is randomly chosen (blind selection and listening) and then the song is played back again.
At the end of the song the listener must decide whether the sound was the same or changed, and then this process is repeated multiple times.
This has the same statistical chance of being correct as the DBT and will produce a result that should be able to be compared to the DBT.
If the results are pretty close then it goes towards validating DBT statistics.
There are some aspects that do intrigue me with the DBT and this is why it really should have its own thread on AVSforum so they can be discussed and explained in greater detail and be one single point of reference.
Some thoughts on the matter.
1. Really we need to see the individual results from any group statistics that are posted from DBT.
This is critical because say the test (theoritical) is done with 10 people, if 2 of those manage 90% and 100% accuracy (a pass), the group DBT statistic can still be reported as a fail due to the others dragging the result down.
This is significant because 1 in 5 did pass the test and highlights that maybe some are more sensitive than others.
This is exactly what happened with the AVReview cable blind testing with 4 listeners (2 passed and 2 failed dragging the statistic down to 70%ish).
2. Ideally any DBT comparison tests should consider the hardware technology and design.
This is critical because it makes more strategic sense to test two products that have differences deemed to be significant, such as one with a cheaper DAC and another with expensive DAC-mechanism-filters-power supply-architecture-etc
This would mean a logical path of seeing say a cheap CD-Player compared with various other players that may have something different, as stated above it could be progression path of comparing more expensive DACs then filters and lastly FPGA DAC-filters, or individual comparison between different power supply solutions, or testing the mechanism to see any difference between a cheap player and one with Philips CD Pro 2 mechanism.
This enables a strategic testing path that enables us to look without bias to see if any one hardware technology/design makes a difference or more realistically a combination.
However there is little point DBT two products with markedly different prices but are using very similar DACs/power supply solution/mechanisms/etc.
The DBT on products based on price is just context, the core aspect must be what is different with regards to the above stated types of differences otherwise what is the DBT looking for in terms of sound and audibility.
And basically this concept can be applied to all other audio product types such as amps.
I personally feel for DBT to be able to move forward this is the level of detail required, and it is this that should be seen to be core to any tests rather than context as it currently seems.
Good old Geoffrey Moore, brought many a joys to my own projects and business strategies