I have a recurring nightmare: I wake up in the middle of the night, in a cold sweat, because I was having the nightmare about…
What’s the Forever Test, you ask? It’s when you set up a test in Google Website Optimizer, or some other testing platform, and for any combination of reasons, the darn thing JUST…WON’T…END! Or worse yet, it runs for way too long, only to bring back inconsequential results, e.g. a 1% increase
Don’t get me wrong–I love GWO, and I love seeing a nice testing win, but seeing a test that just won’t quit is very frustrating to me. I avoid it at all costs. If you’ve ever run into this problem (or maybe you have a Forever Test running right now), read on because I’m going to give you tips on how keep your tests clean, quick, and impactful. Note: because we’re a Google Website Optimizer Authorized Consultant, I’ll be using the vernacular of GWO, but these principles will likely apply to any testing technology.
#1 – Estimate duration: Google provides a free and handy test duration calculator. Bookmark it. Make friends with it. Refer to it often. This tip is my #1 for a reason, it’s the best way to avoid the Forever Test! If you see a duration 1,200 days for example, redesign the test.
#2 – Test things that will actually have an impact: Your tests should be designed with one goal in mind, and that is to see either a big winner, or a big loser. Having a big winner is self-explanatory, but finding a big loser is still useful because it means you’ve found a “lever” that you can now experiment with in order to optimize your site.
The thing you don’t want is to run a test where the variation(s) fare only slightly better or worse than the control. First, this will make the test run longer. Secondly, the results might be, well, not worth much money!
One test I designed, that is still running after 6 weeks, was to test color of pricing on product pages. While it is showing a 1.18% percent increase in conversion rate, there may have been higher priority things to explore first.
#3 – Pick your B2C conversion goal(s) carefully: Unless you get a ton of conversions, think twice about setting your “thank you for purchasing” page as the conversion goal. If you’re testing a change on your homepage, that’s a long way away (many clicks) from your thank you page. Consider tagging the first step of your checkout (e.g. “Added to Cart”) instead of the last if you want to finish your test before you’re a great-grandparent. We talk a lot about micro-conversions as opposed to macro-conversions on this blog, and the principle is key to testing efficiently. Consider testing micro-conversions instead of macro if you want quicker test results.
Also, remember that you can tag multiple pages as the conversion goal. For example, you could run a test on your homepage where visiting any of your category pages would be considered a micro-conversion success point. This is a good approach to take if you’re testing “high in the funnel.”
#4 – Multivariate test with care: Generally speaking, multivariate tests take longer to run. The approach is very insightful, and can be very effective, IF you have lots of traffic and conversions. Due to the nature of multivariate, the permutations in an experiment add up very quickly, and can really add to the duration of a test. Unless you really need to understand how different combinations of elements affect conversions, go with split (A/B) testing. Would you rather have a 10% lift today, or a 12% lift in fifteen weeks? Another trick is to start with A/B testing, get the low-hanging fruit, then move on to multivariate later once you’re rich.
#5 – Run sequential A/Bs instead of A/B/C/D/n: Once you get into testing, it’s easy to generate lots of ideas for variations. In fact, our founders wrote a book called Always Be Testing that has about 64 billion testing ideas in it, so there’s really no excuse for not having test ideas. However, every variation you add makes the test run that much longer, so don’t go overboard. One thing we often recommend is to run sequential A/B tests instead of a longer A/B/C/D/n test. Again, it’s about getting the winning version live on your site as soon as possible so you can make more money. Even if you have 4 versions of a page layout you’d like to test, consider testing them one at a time in sequence. For example: Control vs. Variation1 » Winner vs. Variation 2 » Winner vs. Variation 3 » Winner vs. Variation 4. This “always be challenging the winner” procedure makes sure you get the wins you deserve much more quickly.
I hope this list is added to by our readership, but most of all I hope that this encourages more companies to ‘always be testing,’ AND to stick with it. It can be frustrating at times, but as I detailed in my last post, there’s a lot of competitive advantage to gain if you keep the testing cycle going.