Questions? (877) 643-7244
FutureNow Post
Friday, Apr. 30, 2010 at 9:15 am

Five Tips to Avoid the Forever-Test

By Brendan Regan
April 30th, 2010

I have a recurring nightmare: I wake up in the middle of the night, in a cold sweat, because I was having the nightmare about…

“The Forever Test”!

What’s the Forever Test, you ask?  It’s when you set up a test in Google Website Optimizer, or some other testing platform, and for any combination of reasons, the darn thing JUST…WON’T…END!  Or worse yet, it runs for way too long, only to bring back inconsequential results, e.g. a 1% increase :(

Don’t get me wrong–I love GWO, and I love seeing a nice testing win, but seeing a test that just won’t quit is very frustrating to me.  I avoid it at all costs.  If you’ve ever run into this problem (or maybe you have a Forever Test running right now), read on because I’m going to give you tips on how keep your tests clean, quick, and impactful.  Note: because we’re a Google Website Optimizer Authorized Consultant, I’ll be using the vernacular of GWO, but these principles will likely apply to any testing technology.

Tips For Avoiding the Forever-Test

#1 – Estimate duration: Google provides a free and handy test duration calculator.  Bookmark it.  Make friends with it.  Refer to it often.  This tip is my #1 for a reason, it’s the best way to avoid the Forever Test!  If you see a duration 1,200 days for example, redesign the test.

#2 – Test things that will actually have an impact: Your tests should be designed with one goal in mind, and that is to see either a big winner, or a big loser.  Having a big winner is self-explanatory, but finding a big loser is still useful because it means you’ve found a “lever” that you can now experiment with in order to optimize your site.

The thing you don’t want is to run a test where the variation(s) fare only slightly better or worse than the control.  First, this will make the test run longer.  Secondly, the results might be, well, not worth much money!

One test I designed, that is still running after 6 weeks, was to test color of pricing on product pages.  While it is showing a 1.18% percent increase in conversion rate, there may have been higher priority things to explore first.

#3 – Pick your B2C conversion goal(s) carefully: Unless you get a ton of conversions, think twice about setting your “thank you for purchasing” page as the conversion goal.  If you’re testing a change on your homepage, that’s a long way away (many clicks) from your thank you page.  Consider tagging the first step of your checkout (e.g. “Added to Cart”) instead of the last if you want to finish your test before you’re a great-grandparent.  We talk a lot about micro-conversions as opposed to macro-conversions on this blog, and the principle is key to testing efficiently.  Consider testing micro-conversions instead of macro if you want quicker test results.

Also, remember that you can tag multiple pages as the conversion goal.  For example, you could run a test on your homepage where visiting any of your category pages would be considered a micro-conversion success point.  This is a good approach to take if you’re testing “high in the funnel.”

#4 – Multivariate test with care: Generally speaking, multivariate tests take longer to run.  The approach is very insightful, and can be very effective, IF you have lots of traffic and conversions.  Due to the nature of multivariate, the permutations in an experiment add up very quickly, and can really add to the duration of a test.  Unless you really need to understand how different combinations of elements affect conversions, go with split (A/B) testing.  Would you rather have a 10% lift today, or a 12% lift in fifteen weeks?  Another trick is to start with A/B testing, get the low-hanging fruit, then move on to multivariate later once you’re rich.  ;-)

#5 – Run sequential A/Bs instead of A/B/C/D/n: Once you get into testing, it’s easy to generate lots of ideas for variations.  In fact, our founders wrote a book called Always Be Testing that has about 64 billion testing ideas in it, so there’s really no excuse for not having test ideas.  However, every variation you add makes the test run that much longer, so don’t go overboard.  One thing we often recommend is to run sequential A/B tests instead of a longer A/B/C/D/n test.  Again, it’s about getting the winning version live on your site as soon as possible so you can make more money.  Even if you have 4 versions of a page layout you’d like to test, consider testing them one at a time in sequence.  For example: Control vs. Variation1 » Winner vs. Variation 2 » Winner vs. Variation 3 » Winner vs. Variation 4.  This “always be challenging the winner” procedure makes sure you get the wins you deserve much more quickly.

I hope this list is added to by our readership, but most of all I hope that this encourages more companies to ‘always be testing,’ AND to stick with it.  It can be frustrating at times, but as I detailed in my last post, there’s a lot of competitive advantage to gain if you keep the testing cycle going.

Add Your Comments

Comments (49)

  1. I agree with your opinion. But how many people will understand these tips.

  2. Interesting tips, It has happened to me once or twice and can drive me crazy. Certainly something to keep in mind.

  3. The tips is quit complicated,I have horrible experience also in testing. I gonna try this tips.

  4. I for one enjoyed this post, Brendan. I have a couple of “forever tests” going as we speak.

    Another way I like to avoid these is by integrating the testing platform with Google Analytics to test multiple goal points or revenue per visitor…

  5. I haven’t had the courage, I guess, to start that sort of testing. I think I need more baseline traffic before I can judge up from down.

  6. I enjoyed this post. I definitely vouch for the test duration calculator to avoid the Forever Test

  7. I hate getting stuck in the forever test. I was there once and I changed my whole site around so I would never be there again.

  8. As far as #2 goes, that is why you test multiple angles, and at times web sites. You never know quite how google and conversions will mix it up. I always try a couple different approaches with multiple projects.

  9. I agree with estimate duration as the number 1 tip. If duration is not tested, then you will likely be stuck in the forever test!

  10. Haha oh man that’s funny. It reminds me of engineering when you run a finite element analysis and it takes like the whole weekend to go through billions of calculations. And you come back to work on Monday morning only to find out that the software crashed like Sunday afternoon so there is no data its useless hahaa.

  11. Noted! will definitely remember these tips, especially the first one. To estimate the test’s duration first to avoid the “forever test” really made sense. people often get stuck with their results in GWO because they are disregarding the simplest element of the test.

  12. wow… interesting but trickish. The multivariate approach usually takes a long way but very effective! Thanks for the tips.

  13. Google test duration calculator was a new thing for me.Thanks for this nice tip.

  14. Bullet point #5 is super valuable; its all too easy to get sucked into testing many different ideas at once!

  15. the tips which is given for avoiding the forever test, are very beneficial.
    but implementation is required…..

  16. You can get several +20% boosts on aspects of ANY site.

    This means that if you can’t see an improvement after a week you need to stop the test and try something else.

  17. Good tips, thanks for giving them. test duration calculator is a definitely the best way

  18. I especially appreciate tip #3 as I do quite a bit of conversion testing. Also in Tip #2, did yo say you have a test that’s been running for SIX weeks!, are you serious or is that just hyperbole.

    Thanks for they great info


  19. Is there anywhere that I can find a good tutorial for web optimizer?

  20. I have never heard of Google Website Optimizer. I will give it a try!

  21. Thanks Brendan,I just started using the test duration calculator and it has helped very much.

  22. I must say that the ‘forever test’ syndrome may also lie on the clients side. E.g. not going live because there is something still not perfectly tested, and well, keep testing forever..

  23. Does anyone know of any other program besides Google Website Optimizer? Im curious if theres any higher end programs.

  24. [...] Source:Five Tips to Avoid the Forever-Test [...]

  25. @KEEN: Omniture’s Test & Target is a “high-end” testing platform. Also, you could look into SiteSpect.

  26. Duration may be longer, but that is a reality sometimes with Google and other bigger SE’s. They don’t like blips on the SERP map, so you have to give some things time to settle down.

  27. The forever test haha, yeah but honestly improvements will always be ongoing. I myself am a perfectionist so I don’t know I will ever be satisfied.

  28. I like the idea of continually pitting winners against some other variable to see who will come up on top. always remember to sharpen the saw.

  29. I must agree with an earlier comment, im sure the points and tips raised are valid, but it is hard to understand, maybe a laymens article of google testing would prove useful

  30. @Cameron and others: If you’re interested in building up the basic foundation of GWO knowledge, you may wish to visit

  31. Very good tips and advice to save me tons of time that could otherwise have been avoided. Thanks for these tips! There must be an end date in place for all tests otherwise they will never end!

  32. People often get stuck with their results in GWO because they are disregarding the simplest element of the test.

  33. Nobody would ever want a Forever Test. There simply is to much to do nowadays that you cannot just wait for a test that you have no idea if it will ever finish.

  34. Wow. You have a test that is still going on for six weeks? I do not know if I will design a better test and would probably have to zero in on one goal to avoid the Forever Test.

  35. [...] View full post on Conversion Rate Optimization & Marketing Blog | FutureNow, Inc [...]

  36. The Forever Test, I was wondering what was it when i saw the title, but its useful. Test Duration Calculator is great tool.

  37. Testing is right up there with content when it comes to improving a website or campaigns or pretty much anything really.

    I agree with Point #5 of your post. Getting on the A/B testing is a big deal.

  38. Companies should regularly run a test. There’s a lot of competitive advantage to gain if you keep the testing cycle going. I strongly agree with this advice.

  39. It really is hard to test short term because so many things with google take forever to get into the mix as far as SEO goes. However you can easily test rates within your site cause percentages stay the same.

  40. Great read. Forever is hard to judge, but results definitely need to come fast online.

  41. I find myself guilty of this all the time. I am constantly fiddling with different options, making small tweaks, and what not. I am going to put your advice to use starting now!

  42. Very nice article, I will surely use these tips. However, I have 2 questions about running more experiments on 1 website:

    1) I have read somewhere that if you run more experiments with different conversion pages (for example A: shopping cart to login page B:login to checkout) it will only count one experiment conversion per visitor, so every experiment will take longer that way. Is that true?

    2) Lets assume experiment A is to test 2 versions of css stylesheet for whole website (all pages) and decide which design converts better. Experiment B is shopping cart to login page micro conversion test. How will GWO decide which combination of css/shopping cart page will display? Can´t be the results for those experiments running at the same time wrong by displaying worse version of A and better of B, which can lead to better results for worse version of A?

  43. I find this article funny because I have a Google test running in the background while reading this page. Thanks for the great tips, I will start another test today to try and narrow things down a bit.

  44. I agree with tips #1 and #2. I was just wondering though if there’s a similar program like that of a Google Website Optimizer?

  45. [...] View the original article here Tags: avoid, experiences [...]

  46. On #3, how are micro-conversions valuable if they don’t lead to a sale/revenue/sign-up?

  47. @EAT Club: micro-conversions MUST lead towards a sale/revenue/sign-up. They are still conversions of a smaller degree, or you’re right they aren’t worth measuring. For example, if your funnel conversion rate (view cart » confirmation) is 30%, and you run a test that drives more prospects into your view cart page [micro-conversion], that gives you X number of additional visitors that you will convert at 30%.

  48. Noted! will definitely remember these tips, especially the first one. To estimate the test’s duration first to avoid the “forever test” really made sense. people often get stuck with their results in GWO because they are disregarding the simplest element of the test.

Add Your Comments


Print this Article

More articles from Brendan Regan

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: