- Conversion Rate Optimization & Marketing Blog | FutureNow - http://www.grokdotcom.com -

Does Your Testing Organization Have A Gag Reflex?

Posted By John Quarto-vonTivadar On August 26, 2008 @ 2:52 pm In A/B Testing,Improving Conversion,Presentations,Website Optimization | 4 Comments

gag reflex [1]

Does your organization or company have a gag reflex when it comes to testing? Never mind the medicine to fix a problem, are they even prepared to acknowledge that the problem exists?

Ronny Kohavi from Microsoft’s Experimentation Platform had a short post the other day [2] about Dr. Ign├íc Semmelweis (pronounced “Eeg-natz Shem-mel-vise”, those darn Hungarians!), a 19th century doctor, whose observations (and subsequent testing) of what only a generation later would be understood as cross-patient germ infection, reduced mortality 10-fold when applied. It also ran counter to established medical opinion and Semmelweis was subsequently run out of his hospital.

Today, an organization that does that is euphemistically said to have a Semmelweis Reflex, which Ronny quotes from Wikipedia as “a reflex-like rejection of new knowledge because it contradicts entrenched norms, beliefs, or paradigms”. A related adage for this same idea, one I’m sure you’ve all heard, is “don’t drink your own kool-aid” [Admittedly, even that phrase is three decades old now -- how many readers know the events that gave rise to that reference? And who even drinks kool-aid anymore in this Snapple-VentiChai-RedBull world?]

Now, the lesson is not that anyone with an opinion (and a bullhorn) should be considered a prophet and possibly tomorrow’s saint; no, there really are plenty of kooks in the world. The Semmelweis Reflex is not only reserved for groups who are, hmm, let us say, “ignorant of the facts”. Rather, the Semmelweis Reflex is just as prevalent and possibly more dangerous in a group that considers itself experts in its field — the ones who should know better than to reject new knowledge out of hand — just like those fine well-intentioned doctors at Semmelweis’ Vienna medical research institution.

Does your organization not give testing the weight it deserves because you “know” such-and-such is true? Is your conversion rate maxed out at 2% because “that’s the way it’s always been”? or “that’s what everyone else in my industry gets”? or “I really know what my customers want”? Perhaps you indeed do; if so, testing and re-confirming core assumptions is always healthy.

Ronny’s post (download the PDF he provides) is absolutely worth its quick read (and worth sharing with HiPPOs [3]) and it just might start you thinking. And the next time time an expert — and (gulp!) even an expert at FutureNow — tells you that such and such is true, feel free to question it! Ask for evidence, or better yet — try a small experiment of your own to confirm that idea.

And make “because I say so…” a phrase best reserved for children’s bed-times.

Editors Note: John co-authored Always Be Testing [4] with Bryan Eisenberg and is FutureNow’s Chief Scientist. John worked as an astrophysicist at NASA (on parts of the Hubble Telescope that work). So when John talks about the science of testing, we listen since when a mistake on a NASA multi-variate test occurs the impact can be a lot worse than that of an inefficient “Add-To-Cart” button.


Article printed from Conversion Rate Optimization & Marketing Blog | FutureNow: http://www.grokdotcom.com

URL to article: http://www.grokdotcom.com/2008/08/26/does-your-testing-organization-have-a-gag-reflex/

URLs in this post:

[1] Image: http://www.grokdotcom.com/wp-content/uploads/John_Q/gag.jpg

[2] short post the other day: http://exp-platform.com/semmelweisReflex.aspx

[3] HiPPOs: http://rich-page.com/win-at-web-analytics/win-at-web-analytics-top-7-ways-to-influence-your-hippo/

[4] Always Be Testing: http://www.amazon.com/Always-Be-Testing-Complete-Optimizer/dp/0470290633

Copyright © 2009 FutureNow's GrokDotCom / Marketing Optimization Blog. All rights reserved.