Questions? (877) 643-7244
FutureNow Post
Tuesday, Mar. 21, 2006 at 5:49 pm

AB Testing: Too Little, Too Early?

By John Quarto-vonTivadar
March 21st, 2006

Lately, everywhere you go analytics industry folks are talking about AB Testing. That’s a good sign, since it means the industry is focusing on an overlooked leverage point in their web analytics investment.

But as so often happens, achieving full buzzword compliance has become the goal rather than the means; what lies behind the words is often lost. In this case, “AB testing” – the buzzword – has become a euphemism for plain old “testing”, which, like ordering liver on a first date, may be good for you, but is certainly not sexy. But throw some “AB” in front of “testing” and your dour liver is magically transformed into paté de foie gras.

This is a bit disturbing, especially when you hear people sprinkling the “AB” condiment to add flavor to anything from a focus group (“Hey, did you AB Test the response to the new company logo?”) to the mundane (“Suzie’s lamp is out, can you AB Test the light bulb?”) to the painfully comical (“Honey, let’s AB test the Lord of the Rings Director’s Cut with the Wide-Screen edition!”).

Mixed in there, perhaps lost among the cacophony of buzzword hype, are the ingredients to some real AB testing and with it a future vision of how to achieve its true objective.

What is AB Testing?

AB Testing is based on a simple principle that we’re all familiar with:

  1. compare and contrast alternatives;
  2. based upon measurement, act accordingly.

Let’s say we want to determine whether Nolan Ryan is a better baseball player than Homer Simpson? How should we proceed? First, we might set a metric for what we mean by a “better” baseball player. We can measure evidence in concrete ways, noting the two subjects’ different batting averages or RBIs or the like. What we’re searching for is the right metric—a formula that would lead us to a correct decision. Such a formula is more precisely termed a “fitness function.”….

This is the begining of a 14 page paper John wrote recently about AB testing. We’re still not sure how we will be distributing it but if you are interested in reading it then email me ( jeff at ) and I’ll send you the PDF.

Add Your Comments

Comments (1)

  1. Your guide to all kinds of Lounge Chair Cushions.

Add Your Comments


Print this Article

John is the co-author of the best-selling Always Be Testing and 3 other books. You can friend him on Facebook, though beware his wacky swing dancer friends, or contact him directly at

More articles from John Quarto-vonTivadar

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: