Originally Posted by dinomite
I guess I'm missing something here. You're saying that you can take the subjectivity out of the test, by choosing the right people?
Actually, that is close to correct if you substitute the term "error" for the term "subjectivity". We can minimize the chance that our test gives us an erroneous result (fails to detect a real effect or detects a false effect), even if we are recording subjective data.
My point is that there are methods for doing all this. There is nothing remotely mysterious about it - it is first year undergraduate material in psychology, sociology, many aspects of biology, marketing, political polling etc.
Yes, there is a whole library of methods that allow you to ask whether people can hear a difference between components - and even how big the difference has to be before it can be heard. Sadly, doing quality tests is much harder than doing informal ones. So much easier to use measurements (which we use to "operationally define" the good) or reviewers or our own listening. In fact, a combination of these works pretty well because people keep buying new gear and liking what they buy. Clearly, these people hear a difference.
So it does come down to picking the right people. In making inferences about a population, we need to use formal methods to define the population (people in general, audiophiles, experienced audiophiles, whatever), select the samples, and present the test. Then the uncertainties that Jon raises are minimized.
Or not. We can just realize this is a hobby and have fun with it. Maybe you believe you hear a difference between two brands of speaker cable. Believing that makes you happy. Maybe you could demonstrate the cable sensitivity on a proper blind test or maybe you couldn't. For example, I can't hear cables though I hear amps and preamps pretty reliably on informal tests. So, I use low cost cables and great amps/preamps. Works for me, and I agree with Jon when he implies that the differences that are real are the ones that matter to you. Just don't bash formal testing methods (such as blind testing) unless you understand the theory behind them.