Start Multivariate Testing: 7 Critical Questions
There are several factors that hold people back before they begin A/B or Multivariate testing. If your organization isn’t testing, hopefully we can help you realize that testing isn’t scary. In fact, what our clients have seen once they do start is that NOT testing is what’s really scary. (“We missed out on how much!?” is a common refrain.)
Here are some answers to questions we often get about testing:
Q #1. What tools do we need (aren’t they expensive)?
A. There are various tools you can pay for, including: Optimost, SiteSpect, Verster and others. Or you can use Google’s Website Optimizer for free.
Q #2. What should we test first?
A. Try anything at first, just get used to making a change on your site that you can test. Any change will do so long as you become comfortable with the tool you are using. Here’s a list of 64 things to start your multivariate testing to choose from.
Q #3. Does it take a long time to run these tests?
A. It depends, but it in most cases it shouldn’t. Google Website Optimizer has a nice calculator that helps you understand this. The calculator asks for 5 variables and tells you how long it should take to complete a test; which depends on the number of test combinations, number of page views per day, percent of visitors in who will see the experiment, current conversion rate, and percent expected improvement.
Q #4. Should we be testing thousands of variations?
A. This question illustrates the market’s misunderstanding of testing. For the vast majority of businesses this is more like random testing. You can test thousands of combinations in a multivariate test, but being able to doesn’t mean you should. I’ll explain why the market worries about this further down in this post. But, for now, lets focus on an example. I’ve keep the numbers simple for clarity’s sake, but let’s assume:
Example I (not recommended)
1,000 = Test combinations (the number of page sections and variations in the test)
10,000 = Page views per day
100% = Visitors in experiment (we’ll run the experiment with all our traffic)
2.4% = Current conversion rate (average conversion rate)
20% = Expected improvementThe duration for this test: 34.9 days. (More than a month!)
Example II (recommended):
20 = Test combinations (focus on key drivers)
10,000 = Page Views per day
100% = Visitors in experiment
2.4% = Current conversion rate
20% = Expected improvement (focus on key drivers in the hierarchy of optimization instead of random elements and your expectations should rise)The duration for this test: 0.698 days. (Under a day!)
Q #5. Is it less “scientific” to test less variables?
A. Under the guise of being “scientific,” the companies that originally offered these tools charged on a monthly basis. While they had plenty of experience in managing their software, they had little experience in identifying valuable tests. Plus, they had zero incentive to get quick results while customers paid a monthly fee.
Multivariate testing for the sake of conversion rate optimization should be scientific. However, testing is about improving your business results, not scientific experimentation. Unless you’re running a lab, you’re testing for profit. (No offense, non-profits… Yes, you should be testing too.) Testing only what matters is how to recover opportunity cost.
Q #6. Since time is of the essence, how do I maximize ROI on time?
A. I’ve written several books and hundreds of articles about the variables that matter. Time is money. Don’t waste it by testing which variables matter; rather, invest your time in improving those variables and your understanding of them. Fix the things that hurt your conversions as fast as possible, and make more money today. That’s why you’re reading this, right?
Q #7. How can I quickly learn more about testing?
A. We’ve compiled 7 free resources for first-time testers and do-it-yourselfers. Enjoy!
Related Posts:
Written by:Bryan Eisenberg
[…] Read more of this story… […]
Hi Bryan,
Thanks for the coverage of Multivariate testing, it’s an area that not many people know about just yet and as you say, the uplift in conversions can be very strong!
Just one point - the guys that charge monthly do so because it makes sense for many companies to perform this testing on an ongoing basis. Starting with testing on the homepage then moving to pages deeper within the site or PPC/CPM specific landing pages will give almost a year’s worth of work anyway so it’s more cost effective to enter into a contract than to approach each project separately.
There’s also continuous optimisation technology that monitors conversions then automatically optimises towards the best performing content. For some of our clients, running this on an ongoing basis for their homepage means their best performing content for time of day or time of year is always displayed prominently.
Google’s tool is quite rudimentary and the paid for tools offer a far more full featured product in order to justify their pricing.
You also missed a company; Maxymiser is a European based expert in this field, we have our own blog at http://maxymiser.blogspot.com and our website is at http://www.maxymiser.com
Kind regards,
Alasdair
[…] Start Multivariate Testing: 7 Critical Questions […]
[…] Start Multivariate Testing: 7 Critical Questions […]
Google Optimizer: free tool to increase sign-ups, donations or anything else on your site…
Google has recently released a new product: a testing tool called Website Optimizer to help sites maximize…
[…] http://www.grokdotcom.com/2007/06/12/start-multivariate-testing-7-critical-questions/ […]
Interesting… I have heard mention of multivariate testing before, but am very interested after reading this article.
Grokdotcom does it again.
I everyone …
A friend of mine talk to me about your site (grokdotcom.com), and he told me it has some usefull and valid SEO info.
I allready know google´s tool, but i`ve never heard before of
Optimost, SiteSpect, Verster .
I`ll do a search about the price and advise the company where i work to invest in those tools (if they could effort it)
Thanks for the usefull tips …
Hi Artur,
you can find more information on those tools and also Maxymiser and Offermatica (now Omniture) by simply googling them. I don’t think many will offer a ‘price list’ because the level of engagement will vary depending on what your organisation is looking to achieve from the multivariate testing.
i hope this helps
Kind regards,
Alasdair
Arthur & Alasdair,
We offer website optimization testing as well. In fact, all of our services have testing as an integral component.
Remember, the important part isn’t which tool you use to test — it’s knowing WHAT to test, and how to optimize the site according to visitor motivations.
Hope that helps!
-Editor, GrokDotCom
Hi everyone,
Alasdair, thanks for answering my comment.
Robert, thanks for answering too.
As i said, i`ve mencioned futurenowinc, and yours services, to my boss. Our site, it`s not optimized for sales. I´m trying to show them that investing on a company that offers services like yours the ROI will be garanted.
Sorry for my english, i`m portuguese.
Cumps,
For small online business owners who don’t have a landing page that gets 10,000 page views per day, a fractional factorial (e.g. Taguchi) tool might work out better.
Google’s Website Optimizer can run small Taguchi tests — just use the main effects results instead of the combination report. But it’s difficult to run a test with more than 1000 combinations this way if you don’t have a ton of traffic.
Also, if you aim to improve a metric that’s already getting about a 20% conversion rate, you can confirm a 20% improvement (i.e., going from 20% to 24%) much more quickly than you can if you start with a metric getting a 1% conversion rate.
So if you’re optimizing a sales page getting 1%, you might need up to 60,000 visitors to confirm a 20% improvement (i.e., going from 1% to 1.2%). But going from 20% to 24% can be confirmed in about 1200 visitors.
The key is to find a metric that’s correlated with sales, and which starts out in the “sweet spot” of 10%-50% conversion rate.
There are two nice ways to do this in my experience. Break the page into multiple pages. Or measure something like “time on page”. Then you can measure something (visitors staying > 60 seconds, or visitors clicking to page 2) that starts with about a 20% conversion rate instead of something (sales) that starts with a 1% conversion rate.
To the extent that these other metrics are correlated with sales, you can speed up your testing by a factor of 10 or more.
For those who are interested, the Split Test Accelerator provides a way to run large tests with very little traffic. It offers faster tests than Google’s Website Optimizer, and it doesn’t cost hundreds or thousands per month like most hosted Taguchi solutions.
All of the tools mentioned in this post are good options, AFAIK. As mentioned before, the ability to find good factors to test is more important than the tool you use to a large degree. But some tools are out of some SBO’s budgets, and others take too long for sites with low traffic.