Axiom Home Page Forums General Discussion Technical Questions Hear No Evil: another essay

I have noticed the pervasive faith that people in this forum have in double-blind testing of audio equipment. This is interesting to me, as I have usually associated audiophiles with faith in the subjective. I mean honestly, I'm surprised to see people who love Axiom speakers yet believe that almost all the other components are barely differentiable. How do the skeptics justify or prove the excellent performance of their speakers? Apparently any personal observations are so inaccurate and variable as to be useless.

Does that mean that there are skeptics here who bought speakers based purely on frequency response curves and other measurements? Or is it because of trust in Axiom's use of double-blind testing to confirm performance traits? But that brings me to my point: double-blind testing is useful at getting gross statistical information about listener reactions, but it has severe limitations. An argument commonly put forth in Stereophile is: "Audio equipment is for listening, not comparing." That is to say that most people simply want to enjoy the music through their equipment: that is the only goal.

See no evil
The implication would then be: if the equipment sounds enjoyable enough to justify the price, then it doesn't matter if the measurements are terrible. What would skeptics do in this situation? Conclude that they were under some delusion, that their ears were going bad? Similarly, if double-blind tests indicate that "no difference" is heard, will that prevent you from making a purchase? Welcome to low-fi, mass produced audio equipment for the masses. Ignorance is bliss, and much less expensive too!

Because blind testing is subjective, I will tend to agree generally with the results. What's dangerous to me is to come to some conclusion when no difference is detected. Or when no study was ever done, as with most audio equipment. That does not prove anything; rather, it indicates a failure to gather decisive data. I believe that most pieces of audio gear are audibly distinguishable, but it depends entirely on the experience of the listener. Once you have listened critically to various equipment, and once you know what to listen for, you can vastly improve your signal to noise ratio.

So what if the experiment subjects don't share my scrutiny of the mid-treble region, what if they prefer a bass hump at 100Hz, what if they don't know the recording, or what if they just didn't listen long enough? I don't pick up on more subtle characteristics until listening to dozens of different CDs on the same equipment. Likewise, what if the measurements taken don't cover the traits I'm interested in? Namely, my musical enjoyability. I don't "enjoy" a response curve, I don't "savor" high impedances, nor do I cry because of a speaker resonance.

Chocolate
Example: in a psychology class on perception, we set up blind taste tests of two different chocolate bars: Hershey Special Dark versus Lindt dark chocolate. Most students could not consistently distinguish between them, but I had a 100% hit rate and only one or two false positives. Experiments are great, but there's an infinity of variables to control for, and you're limited to a) what you can anticipate, b) what's practical, and c) the subjects' abilities. Theory will always fall short of reality because it is a synthetic reconstruction. Trying to measure all aspects of audio equipment's performance is like trying to make a perfect circle on the beach using pebbles.

-Cooper

Cooper,

Some interesting points you bring up. However, these sorts of posts are invitation for everybody's $0.02, so...

A lot of the people posting on this board are scientists of one form or another, such as engineers and doctors. As such, they (we...I'm one myself) have a predisposition to favor hard science over subjective descriptions we might consider "flights of fancy". It's the deeply entrenched skepticism of scientific/philosophical inquiry, and if you've thought that way for 30 years, it's pretty darn hard to change your approach to things. And heck...if speaker design can be described in measurable numbers a-la the NRC curves, a scientist is drawn like a horse to water.

So what happens when reviewers name a power cable product of the year, despite no statistically measurable difference? The scientist dismisses it...at best, it's expectation bias, at worst it's sheer nonsese.

Much as I love watching 2x6spds raise the wrath of Semi and Chesseroo by claiming megabuck speaker wire makes a difference, the truth is there's nothing wrong with believing in the subjective. It's just that with us fuddy-duddy scientists, it'll never happen until that p-value hits less than 0.05

Hi Cooper!

I can't speak on the technical side of this as I'm still pretty much a newbie when it comes to audio equipment--and I don't know the technical/engineering side like many others here do. But like FHW, I am also a scientist by training, and I have no problems with wanting to see some support for claims. I also feel strongly that a critical philosphical element of any kind of objective, scientific experimentation is disciplined attention to the null hypothesis...the "starting point" that says there is no difference (whether it be the perceived efficacy of a pharmaceutical vs. an existing one or a placebo, or the transmission qualities of different kind of speaker wire). When you say "What's dangerous to me is to come to some conclusion when no difference is detected", I think just the opposite...until a difference is detected it's dangerous to think one exists.

I know we're not talking about earth-shattering issues here--it's auditory equipment not cancer drugs, and life is too short for us not to be enjoying it, including joy in listening to music. But at the same time if someone makes a claim (whether they're a manufacturer promoting their product or otherwise) I'm always interested to see if their claims hold up. As an example, I'd love to see if folks could distinguish between similar-spec'd receivers given the same accoustic setup...I think it'd be really interesting and answer some questions (maybe it's been done already). In any case, this kind of information helps me make better (and more informed) purchase decisions as a consumer, and IMO also ultimately promotes better engineering vs. a heavy reliance on marketing (you know who's the KING of THAT).

Sorry if I missed your point Cooper (and I'm stepping off my improvised soapbox). I agree with you that ultimately it's up to each person to make their own decisions; I'll just always welcome the ol' p-values when doing so.

Take care all.

Larry

Cblake

I think your meditation is profound. I particularly like your analogy of drawing a perfect circle with pebbles with the relationship between theory and reality.

I'm not a scientist but have faith in reason, if that doesn't strike those without a poetic organ as too funny.

I have no idea what an individual electron, if there is such a thing, or a gaggle of them, 'experience' as they pass through wires of different properties. If a 'scientist' can say, with confidence, that every factor bearing on such a flow can me objectified, identified, and measured, then what's left but measurement of all relevant factors? But, our day to day perception is on a different level of magnitude than the phenomenon under inspection - the flow of electrons through power Cord A and B, or the modulated output from an amplifier through speaker wires A and B.

Seems to me that a scientist can say, that as to those factors which we can identify and measure, we find there are or are not significant differences between the flow in wire A or B.

However, it may be, that if a scientist could 'ride' an electon, his/her point of view would be substantially different. In particular, the resolution of detail would open a potential universe of variables which are not perceivable from our point of view, or maybe just not obvious, or discovered yet.

What if a scientist could only measure the voltage of the signal entering the wire and the voltage of the signal leaving the wire? Armed with his single variable, but confident that it is sufficient to discern any difference in performance of various test wires, such a scientist would have all the data he would need for his tests and conclusions.

Such a flatlander could announce with certainty, that there is no difference between wire A and wire B in terms of the voltage entering and leaving the subject wires.

More than resistance and impedence, etc., what other variables do we really need to quantify in order to characterize the performance of wire? Ask a quantum audio scientist from the late 22nd century, there may be more variables under heaven than Horatio's fine science can discern ...

Which leaves the subjective observer. Who could put confidence in such a wet and messy piece of equipment? Why would anyone think that such a lump of protoplasm could possibly discern differences in the effect of various speaker wires on the flow of electrons, 'things' we understand so well, when our sophisticated multimeters indicate no discernable difference in what these devices can measure?

I don't use megabuck cables. I use Bob Crump design power cords and like the modestly priced but weirdly thin Mapleshade Double Golden Helix Plus speaker wires for my tube amp driven stereo system. Can I hear a difference between the lamp cord I used before? I think so. Can you?

I can hear a difference between different amps and processors. Can you?

I can hear differences between different speakers. Can you?

I think all the Scientists should be less Martha Stewart and be more BAM! And take it up a notch!!!
Scientists have not eaten that "pork fat" since its statistacally bad for them.
They don't really know how GREAT it tastes.
Oh wait...oh wait...did I hear a scientist just say "the pork fat releases the endorphines and that is why people feel good and it is not actually good but ....

figures they would say that....

Nod and smile, nod and smile...

*backs away slowly*

Good points all around, and I appreciate the humor. A laugh goes a long way, no? The number of perspectives that we have, just between different human beings, is immense, not to mention going outside the human and into the theoretical.

I would still like to respond to the null hypothesis idea. I am familiar with this: in statistics, you begin with a null hypothesis that there is no difference between two choices. Only once you have exceeded your margin of error, usually two standard deviations or ~95% certainty, can you claim that there is a statistically significant difference.

However, I stand by my original claim: when the difference is not statistically significant, then the experiment is a failure. You go into an experiment with the purpose of discovering the magnitude and direction of the suspected difference between two variables. If you are "unable to reject the null hypothesis," it means just that. You can never prove a null hypothesis, only disprove. That's why it's called "null:" it has no value.

The funny thing is that, in general, I am an extremely analytical and scientific person. I probably analyze and contemplate the Stereophile graphs more than most readers, and I personally would like my amps and sound sources to have high power and low distortion.

Let me put it this way: I think it's important for us to increase the number of pebbles for our circle, so that we can better understand and better replicate high fidelity. I just think it's more practical and efficient to allow ourselves to be guided by both personal observation and scientific testing. After all, the most profound scientific results come either unexpectedly or from creative hypotheses. Personal observation can give birth to new alternative hypotheses. New hypotheses can lead to new and better forms of testing and measurement.

The bottom line to the listener is to listen and enjoy. Let the listener within inspire the scientist within.

-Cooper

More pebbles, more tolerance, more accurate reproduction of music.

LOL...loved the "nod and smile, back away slowly"...! Sage advice!!

I'm not trying to put down the value of subjective satisfaction...you oughta have what you love, and if you think it works that's great. But at the same time, personally, I LIKE knowing if $300 speaker cables make a difference (or not) over $30 ones. If blind designs show people can't tell the difference between the two, many will still buy the more expensive ones to get the "better" cables and feel better about their system...and that's great. Whatever trips your trigger; for me, that info is helpful because it can save me some money.

At the risk of seeming to throw down a gauntlet (and I'm NOT) I just have to say I fundamentally disagree with the idea that:

"when the difference is not statistically significant, then the experiment is a failure. You go into an experiment with the purpose of discovering the magnitude and direction of the suspected difference between two variables."

IMO, truly objective researchers may HOPE for a difference, but have to ASSUME going in that there isn't one. Analogous to innocent until proven guilty. And in fact many times not finding statistically significant findings IS significant and meaningful...although perhaps not an end result desired by pharmaceutical companies, audio accessories manufacturers and the like.

To sum it up, although it'd stop a lot of back and forth (fun) discussion about speaker "break in", speaker wire, receiver "brightness", etc., all I'm saying is that I'd like to see any studies where people have done blind comparisons (e.g., all factors controlled except for brand of receiver) to see if there IS an auditory difference. If not, well, the placebo effect is real, but I don't necessarily want to pay for it.

BTW, got my Axioms yesterday and WOW...very nice...the black oak is MUCH nicer than I anticipated. Can't wait to set them up...going to do it this weekend.

Take care all.

OK, I'm going to regret this later, but I've just got to weigh in on the "null result" thing. If more people/scientists actually accepted the "null result" as a valid result (or the failed hypothesis, etc), I think that science would be far more advanced than it is today. Far too many researchers (and I'm not saying all, or most, but some is too many) fudge the data to get results, and it seems (I'm not a scientist, so feel free to smack me down) that there is a culture of not publishing "failed" experiements. In reading quasi-scientific publications (Smithsonian, National Geographic, Natural History), it is often refreshing to see something like "Our hypothesis was proved wrong," in the text.

Just my two bits.