Questions? (877) 643-7244
FutureNow Post
Friday, Jan. 28, 2011 at 8:23 am

What to Do When You See a Test Failing

By Natalie Hart
January 28th, 2011

If you test, whether it be A/B or Multivariate, chances are you’ve seen at least one test that has failed. We’d all like to get a 100% success rate, but testing itself is conducted because there are visitor behaviors we are incapable of predicting. Even with 10+ years of experience in behavior-based conversion rate optimization, we here at FutureNow are sometimes wrong. So, the question is, when you see that your test is failing, what do you do?

Just like there are patterns in successful tests, there are patterns in failing tests as well. But the place you should always start is at your hypothesis or research question. You’ve heard us preach the importance of having a hypothesis when creating your tests, and they’re just as important when getting to the bottom of why a test is failing.

Tests fail for 3 primary reasons:

  1. you’re not testing a drastic enough change
  2. your hypothesis is based off of old or incomplete data
  3. you’re testing something that had little affect on visitor behavior

Chances are, if you go back to your research question, you’ll receive some insight as to where your test might have failed. Maybe this will lead you to re-working your research question, or perhaps lead you to a different hypothesis altogether. Either way, understanding WHY your test is failing is just as important as what you do after.

Above are two screen shots of real tests run by clients. Both are failing, but they are very different. The one on the left is being beaten by the original, and has been consistently so since early on in the test.  The one on the right has been going back and forth for over a month and we’re currently at a dead tie.

Many of you will be surprised to hear that of the two test results, the one I’d prefer to see is the one on the left. This is because I know exactly what to do – end the test. The consistency of the tests results indicate that this is a failed test and we need to go back to the drawing board about our hypothesis. With the test on the left, we can still draw knowledge about the visitors from the test data, and use that to either refine our hypothesis, or draw up a new one.

The test on the right is much more problematic. Not only doesn’t it give us clear results, but it doesn’t leave us with much knowledge to use to refine our hypothesis.  I’d still suggest to end this test as well.  The difference is, ending it puts us back at sqaure one: we still are no closer to understanding who our visitors are and what affects their behaviors.

Tests should reach completion within 60 days, ideally 30 days. Not sure how to estimate the duration? Use the free test duration calculator tool provided by Google to figure it out. If your test is going to take longer than 60 days, it’s not the right test to be running.

But what if your test is trending negatively after only a week? While there’s no steadfast rule, I always encourage my clients to run their tests for at least 2 weeks. This will give us a better impression of visitor behavior, and many times, things begin trending in a more positive direction. Having said that, letting a negatively trending test run for an additional week is not always the best choice in all cases.

If you have a gaggle of tests ready and the one trending in the failing direction doesn’t have as potentially significant an impact as some of the other tests in your pipeline, you may consider pausing the negatively trending tests in favor of launching a new test. Know what you have in your testing cue. At the end of the day you should always be testing whatever will give you the most significant results first.

Add Your Comments

Comments (21)

  1. That makes a lot of sense but I have to admit that I had never though of it like that. I was of the assumption that if test results weren’t clear, then you would need to test more or longer.

  2. When this happens I figure the changes to the pages aren’t big enough to matter, so stay with the original and try another test, pretty simple. :)

  3. I think this article is full of sound advice in terms of test length and visual results for a failing test, but I think some specific examples of variables generally not worth testing or examples of insignificant changes would have been interesting as well.

  4. Good advice on running the test for at least 2 weeks. Sometimes, when you only look at a week’s data it can be misled by day over day effects within the week. Running the data for at least two weeks can help mitigate that, even if the results are trending strongly during the first week.

  5. Natalie,

    That’s a terrific blog-post. Every client that is in the web-space is very conscious these days about conducting the right kind of tests for their conversion funnels.

    Be it A/B tests or be it multivariate testing, you definitely need that some stage of your activities.

    The only point that becomes a challenge is allocating the right time-frame.

    Typically for medium organizations, we think it is a big challenge to allocate a time period of 30 days in-order to reach completion. In other words, on paper everything is ok but real problem comes when you start your tests.

    What do you think?

  6. Isn’t there a margin of error with respect to both parameters, so there would really be bands rather than lines for the data? Might the graph on the right show the two lines are actually within the same band, so the slight variations may not really be significant?

  7. Tests fail for 3 primary reasons.if test results weren’t clear, then you would need to test more or longer.

  8. I love the test duration calculator! Very helpful. Great insight on the usefulness of test data. I think many of us would prefer the test data on the right – but, I guess we’d be wrong!

  9. I agree with Finally. The test duration is very helpful.

    I’m sure that the three reasons do not apply to me. What can be happening?

  10. If I don’t have success for the third test, I will try again, again and again :)

  11. I, personally, see failing as a sign to try again and again!

  12. when you are failing test then you should testing again and again.I was of the assumption that if test results weren’t clear,then you would need to test more or longer.

  13. There is nothing wrong in failing as long as you know there’s always chances for you to improve it. Remember that there is always a second chance.

  14. When i fall down i will stand up. When i fall down again i will stand up again. if i fall down million times i will stand up million times. we only learn from our mistakes

  15. I, personally, see failing as a sign to try again and again!

  16. Having a time limit on testing is essential as it brings some resiliance to re-testing and new testing. Ive seen so many phd chem students spend the best part of a year relying on the results of one primary test and then it fails to deliver and they crumble. So I agree with you Nat, that resilience and constant turnover is essential in testing.

  17. Test, Test and Test some more is my motto!

  18. “Tests should reach completion within 60 days, ideally 30 days.”

    Oh dear. We have several (much) smaller business clients who are always looking to continually improve their websites. Their conversion rates and traffic typically don’t allow their tests to be completed within 60 days (much less 30 days).

    In these situations though, we believe that having to spend a little more time running tests for these (much) smaller clients is better than running no tests at all.

  19. @Colorado Business Websites – we tend to agree: testing is still worth it. One thing you could try to help the test along is to do a 90/10 traffic split to the new variation page/old control page, instead of the typical 50/50 traffic split.

  20. I agree! There is no reason for you to stop when you don’t have success at the first 3 tests, just try until you succeed.

  21. very interesting, I have implemented test you have mentioned just before 1 week and whola ! I got sucess. Thanks mate.

Add Your Comments


Print this Article

Natalie is a Persuasion Analyst with FutureNow.

More articles from Natalie Hart

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: