There are several factors that hold people back before they begin A/B or Multivariate testing. If your organization isn’t testing, hopefully we can help you realize that testing isn’t scary. In fact, what our clients have seen once they do start is that NOT testing is what’s really scary. (“We missed out on how much!?” is a common refrain.)
Here are some answers to questions we often get about testing:
Q #1. What tools do we need (aren’t they expensive)?
A. There are various tools you can pay for, including: Optimost, SiteSpect, Verster and others. Or you can use Google’s Website Optimizer for free.
Q #2. What should we test first?
A. Try anything at first, just get used to making a change on your site that you can test. Any change will do so long as you become comfortable with the tool you are using. Here’s a list of 64 things to start your multivariate testing to choose from.
Q #3. Does it take a long time to run these tests?
A. It depends, but it in most cases it shouldn’t. Google Website Optimizer has a nice calculator that helps you understand this. The calculator asks for 5 variables and tells you how long it should take to complete a test; which depends on the number of test combinations, number of page views per day, percent of visitors in who will see the experiment, current conversion rate, and percent expected improvement.
Q #4. Should we be testing thousands of variations?
A. This question illustrates the market’s misunderstanding of testing. For the vast majority of businesses this is more like random testing. You can test thousands of combinations in a multivariate test, but being able to doesn’t mean you should. I’ll explain why the market worries about this further down in this post. But, for now, lets focus on an example. I’ve keep the numbers simple for clarity’s sake, but let’s assume:
Example I (not recommended)
1,000 = Test combinations (the number of page sections and variations in the test)
10,000 = Page views per day
100% = Visitors in experiment (we’ll run the experiment with all our traffic)
2.4% = Current conversion rate (average conversion rate)
20% = Expected improvement
The duration for this test: 34.9 days. (More than a month!)
Example II (recommended):
20 = Test combinations (focus on key drivers)
10,000 = Page Views per day
100% = Visitors in experiment
2.4% = Current conversion rate
20% = Expected improvement (focus on key drivers in the hierarchy of optimization instead of random elements and your expectations should rise)
The duration for this test: 0.698 days. (Under a day!)
Q #5. Is it less “scientific” to test less variables?
A. Under the guise of being “scientific,” the companies that originally offered these tools charged on a monthly basis. While they had plenty of experience in managing their software, they had little experience in identifying valuable tests. Plus, they had zero incentive to get quick results while customers paid a monthly fee.
Multivariate testing for the sake of conversion rate optimization should be scientific. However, testing is about improving your business results, not scientific experimentation. Unless you’re running a lab, you’re testing for profit. (No offense, non-profits… Yes, you should be testing too.) Testing only what matters is how to recover opportunity cost.
Q #6. Since time is of the essence, how do I maximize ROI on time?
A. I’ve written several books and hundreds of articles about the variables that matter. Time is money. Don’t waste it by testing which variables matter; rather, invest your time in improving those variables and your understanding of them. Fix the things that hurt your conversions as fast as possible, and make more money today. That’s why you’re reading this, right?
Q #7. How can I quickly learn more about testing?
A. We’ve compiled 7 free resources for first-time testers and do-it-yourselfers. Enjoy!