Questions? (877) 643-7244
FutureNow Post
Tuesday, Jun. 12, 2007 at 12:40 pm

Start Multivariate Testing: 7 Critical Questions

By Bryan Eisenberg
June 12th, 2007

There are several factors that hold people back before they begin A/B or Multivariate testing. If your organization isn’t testing, hopefully we can help you realize that testing isn’t scary. In fact, what our clients have seen once they do start is that NOT testing is what’s really scary. (“We missed out on how much!?” is a common refrain.)

Here are some answers to questions we often get about testing:

Q #1. What tools do we need (aren’t they expensive)?

A. There are various tools you can pay for, including: Optimost, SiteSpect, Verster and others. Or you can use Google’s Website Optimizer for free.

Q #2. What should we test first?

A. Try anything at first, just get used to making a change on your site that you can test. Any change will do so long as you become comfortable with the tool you are using. Here’s a list of 64 things to start your multivariate testing to choose from.

Q #3. Does it take a long time to run these tests?

A. It depends, but it in most cases it shouldn’t. Google Website Optimizer has a nice calculator that helps you understand this. The calculator asks for 5 variables and tells you how long it should take to complete a test; which depends on the number of test combinations, number of page views per day, percent of visitors in who will see the experiment, current conversion rate, and percent expected improvement.

Q #4. Should we be testing thousands of variations?

A. This question illustrates the market’s misunderstanding of testing. For the vast majority of businesses this is more like random testing. You can test thousands of combinations in a multivariate test, but being able to doesn’t mean you should. I’ll explain why the market worries about this further down in this post. But, for now, lets focus on an example. I’ve keep the numbers simple for clarity’s sake, but let’s assume:

Example I (not recommended)
1,000 = Test combinations (the number of page sections and variations in the test)
10,000 = Page views per day
100% = Visitors in experiment (we’ll run the experiment with all our traffic)
2.4% = Current conversion rate (average conversion rate)
20% = Expected improvement

The duration for this test: 34.9 days. (More than a month!)

Example II (recommended):

20 = Test combinations (focus on key drivers)
10,000 = Page Views per day
100% = Visitors in experiment
2.4% = Current conversion rate
20% = Expected improvement (focus on key drivers in the hierarchy of optimization instead of random elements and your expectations should rise)

The duration for this test: 0.698 days. (Under a day!)

Q #5. Is it less “scientific” to test less variables?

A. Under the guise of being “scientific,” the companies that originally offered these tools charged on a monthly basis. While they had plenty of experience in managing their software, they had little experience in identifying valuable tests. Plus, they had zero incentive to get quick results while customers paid a monthly fee.

Multivariate testing for the sake of conversion rate optimization should be scientific. However, testing is about improving your business results, not scientific experimentation. Unless you’re running a lab, you’re testing for profit. (No offense, non-profits… Yes, you should be testing too.) Testing only what matters is how to recover opportunity cost.

Q #6. Since time is of the essence, how do I maximize ROI on time?

A. I’ve written several books and hundreds of articles about the variables that matter. Time is money. Don’t waste it by testing which variables matter; rather, invest your time in improving those variables and your understanding of them. Fix the things that hurt your conversions as fast as possible, and make more money today. That’s why you’re reading this, right?

Q #7. How can I quickly learn more about testing?

A. We’ve compiled 7 free resources for first-time testers and do-it-yourselfers. Enjoy!

Add Your Comments

Comments (17)

  1. [...] Read more of this story… [...]

  2. Hi Bryan,

    Thanks for the coverage of Multivariate testing, it’s an area that not many people know about just yet and as you say, the uplift in conversions can be very strong!

    Just one point – the guys that charge monthly do so because it makes sense for many companies to perform this testing on an ongoing basis. Starting with testing on the homepage then moving to pages deeper within the site or PPC/CPM specific landing pages will give almost a year’s worth of work anyway so it’s more cost effective to enter into a contract than to approach each project separately.

    There’s also continuous optimisation technology that monitors conversions then automatically optimises towards the best performing content. For some of our clients, running this on an ongoing basis for their homepage means their best performing content for time of day or time of year is always displayed prominently.

    Google’s tool is quite rudimentary and the paid for tools offer a far more full featured product in order to justify their pricing.

    You also missed a company; Maxymiser is a European based expert in this field, we have our own blog at and our website is at

    Kind regards,

  3. [...] Start Multivariate Testing: 7 Critical Questions [...]

  4. [...] Start Multivariate Testing: 7 Critical Questions [...]

  5. Google Optimizer: free tool to increase sign-ups, donations or anything else on your site…

    Google has recently released a new product: a testing tool called Website Optimizer to help sites maximize…

  6. Interesting… I have heard mention of multivariate testing before, but am very interested after reading this article.

    Grokdotcom does it again.

  7. I everyone …

    A friend of mine talk to me about your site (, and he told me it has some usefull and valid SEO info.

    I allready know google´s tool, but i`ve never heard before of
    Optimost, SiteSpect, Verster .

    I`ll do a search about the price and advise the company where i work to invest in those tools (if they could effort it)

    Thanks for the usefull tips …

  8. Hi Artur,

    you can find more information on those tools and also Maxymiser and Offermatica (now Omniture) by simply googling them. I don’t think many will offer a ‘price list’ because the level of engagement will vary depending on what your organisation is looking to achieve from the multivariate testing.

    i hope this helps

    Kind regards,

  9. Arthur & Alasdair,

    We offer website optimization testing as well. In fact, all of our services have testing as an integral component.

    Remember, the important part isn’t which tool you use to test — it’s knowing WHAT to test, and how to optimize the site according to visitor motivations.

    Hope that helps!

    -Editor, GrokDotCom

  10. Hi everyone,

    Alasdair, thanks for answering my comment.

    Robert, thanks for answering too.

    As i said, i`ve mencioned futurenowinc, and yours services, to my boss. Our site, it`s not optimized for sales. I´m trying to show them that investing on a company that offers services like yours the ROI will be garanted.

    Sorry for my english, i`m portuguese.


  11. For small online business owners who don’t have a landing page that gets 10,000 page views per day, a fractional factorial (e.g. Taguchi) tool might work out better.

    Google’s Website Optimizer can run small Taguchi tests — just use the main effects results instead of the combination report. But it’s difficult to run a test with more than 1000 combinations this way if you don’t have a ton of traffic.

    Also, if you aim to improve a metric that’s already getting about a 20% conversion rate, you can confirm a 20% improvement (i.e., going from 20% to 24%) much more quickly than you can if you start with a metric getting a 1% conversion rate.

    So if you’re optimizing a sales page getting 1%, you might need up to 60,000 visitors to confirm a 20% improvement (i.e., going from 1% to 1.2%). But going from 20% to 24% can be confirmed in about 1200 visitors.

    The key is to find a metric that’s correlated with sales, and which starts out in the “sweet spot” of 10%-50% conversion rate.

    There are two nice ways to do this in my experience. Break the page into multiple pages. Or measure something like “time on page”. Then you can measure something (visitors staying > 60 seconds, or visitors clicking to page 2) that starts with about a 20% conversion rate instead of something (sales) that starts with a 1% conversion rate.

    To the extent that these other metrics are correlated with sales, you can speed up your testing by a factor of 10 or more.

    For those who are interested, the Split Test Accelerator provides a way to run large tests with very little traffic. It offers faster tests than Google’s Website Optimizer, and it doesn’t cost hundreds or thousands per month like most hosted Taguchi solutions.

    All of the tools mentioned in this post are good options, AFAIK. As mentioned before, the ability to find good factors to test is more important than the tool you use to a large degree. But some tools are out of some SBO’s budgets, and others take too long for sites with low traffic.

  12. Using A/B testing is a very helpful tool. This helps not only the business or person who owns the sit, but the people who visit as well. Keeping your website maintained and up to date will increase sales and traffic. Using multivariate testing you use more examples. This has the potential of confusing some people. That’s why most people are comfortable with A/B testing.

  13. Through the magic of Multivariate experiments, the possibilities are virtually limitless. You can rapidly learn the best combination of page elements- things like your headlines, product images, and price points, to maximize sales or lead flow. Stop guessing what the ideal design for your page is because you don’t have to. Through Multivariate experiments, you can learn if the picture of the girl with the white background, or the man with the blue background works better. Find out if the long form with detailed instructions works better than the short form with short instructions.

  14. unfortunately google has changed many of their webmaster tools for the worse

  15. Thanks a lot for your question.Really very helpful

  16. [...] Start Multivariate Testing: 7 Critical Questions – A set of Q&A regarding how to get started with multivariate testing. [...]

Add Your Comments


Print this Article

Bryan Eisenberg, founder of FutureNow, is a professional marketing speaker and the co-author of New York Times and Wall Street Journal bestselling books Call to Action and Waiting For Your Cat to Bark and Always Be Testing. You can friend him on Facebook or Twitter.

More articles from Bryan Eisenberg

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: