Originally Posted by krabapple
Speaking of research papers, Sean (if I may call you that), what are your thoughts on this review of potential pitfalls of 'MUSHRA' and quality-evaluation-type blind tests, versus blind difference tests, by Zielinsky, Rumsey & Bech?http://www.aes.org/e-lib/browse.cfm?elib=14393
Yes, I am familiar with this paper. It is a review of well-known biases related to any perceptual experiment. Poulton wrote a book in 1989 called "Biases in Quantifying Judgements"
that covers this topic very well.
Most of these biases can be either eliminated, controlled or accounted for in the test design, and/or in the statistical analysis and interpretation of results. They talk about visual biases, which are easily controlled by doing the test double-blind (we wrote a paper
on the effects of sighted vs blind loudspeakers tests, which as far as I know, is the only paper like it).
Where I part company from this paper, is what I feel is an overly paranoid, one-sided (dare I say biased), hasty view about preference (affective) tests. Zielinski, in particular, has gone on record eschewing preference tests because he feels preferences are too subjective, unreliable and give relative ratings--not absolute ones. In some cases that is true, but I've published research that shows listeners can give very reliable preference judgments of loudspeakers, particularly if you use trained listeners. He chooses to ignore that research perhaps because it doesn't suit his argument.
Humans are very subjective and affective beings. Consumers make purchase decisions largely based on affective judgments. If you avoid trying to measure them, then you are not getting to the bottom of the truth.