Questions? (877) 643-7244
FutureNow Post
Monday, Jun. 2, 2008 at 10:21 am

Give Yourself a Winning Chance with Testing

By Holly Buchanan
June 2nd, 2008

Image of Google Website OptimizerBryan Eisenberg did a fabulous post on how to get buy-in for conversion rate optimization. It’s a must read for anyone who is trying to get their company or client to do optimization testing.

I’ve heard from some folks who got initial buy-in, but whose companies quickly lost the stomach for testing. They ran some tests that did well, but they ran some tests that actually hurt conversion. The tests that hurt conversion are especially painful for smaller companies, where that lost revenue is felt directly by the business owner.

So what can you do to keep morale high and commitment to testing firm? What can you do it increase your chances for getting the best results from your testing? There’s a wonderful article by Lily Chiu at Omniture called “How to Make Testing Successful.”

There’s a lot of great advice in Lily’s post, but one bit really hit home with me:

I’ve seen some companies fail because they try to limit risk by treating testing as project-based task that just needs to be executed once per quarter. By doing so, they actually increase risk because the chances of testing being looked upon favorably decrease with every test that doesn’t generate lift. By limiting the number of tests you run, you limit the number of tries you get to knock a test out of the ballpark.

Here are a few more points to add to the list:

• Give yourself more chances to win. Get commitment to run more than just a a few tests. The more tests you run, the better your chance to create a real winner.

• Any tests that hurt conversion are only temporary. You only hurt conversion for the short period of the test. Once you get valid results, you can revert back to the control and immediately stop any losses. But tests that increase conversion? Those are the tests that keep on giving. You will enjoy those gains, not just temporarily, but for months, and years to come. Even a small increase can have a big impact on the bottom line when you enjoy those gains over long periods of time.

There is no such thing as failure, only feedback. Now, I know that business owners are probably thinking, “Yeah, easy for you to say. It isn’t your wallet that’s affected.” But look at the opportunity cost of NOT doing testing. You are left guessing which changes to make to your site, or you may not be making any changes at all. How much MORE money could you be making? You’ll never know.

Testing is one of the best ways to gain real insight. You learn something from every single test. The combination of that learning IS going to result in gains, but it doesn’t always come in the first few tests. So hang in there. Don’t get discouraged. Sometimes you knock it out of the ballpark in the fest few tries, sometimes it takes a little longer. The more tests you run, the better your chances for success.

Hint: Want to get better create better tests? Personas can help you know what’s worth testing.

Add Your Comments

Comments (11)

  1. One issue I have with split testing is the way results can almost reverse halfway through!

    In one test, version B was getting a 25% conversion rate, while version A had 0%

    Two days later, Version A has all but caught up, so the different between the two is barely discernible.

    I guess the only think to do is give it more time as environmental factors could influence it??

  2. Holly – glad you found my post useful! Your additional points are great. I agree that tests that don’t provide lift should not be seen as failures. The definition of “success” in a test should take both the lift and learnings into consideration. For example, if you learn that having models in your product images doesn’t affect conversion or revenue, that’s a huge take-away that allows you to skip on paying for models in the future! Also, imagine what happens if you don’t test and go solely on gut. The probability of pushing a “loser” out into production is probably pretty likely.

    Matt – you make a great point about how important it is to allow your data to normalize over time. Depending on traffic and environmental factors such as time of day and day of week, I’ve seen results change pretty dramatically. In your example, you might want to see if version B is the best performer for new visitors and version A is the best for return visitors.

  3. “you might want to see if version B is the best performer for new visitors and version A is the best for return visitors.”


    just feeling my way with straight A/B, – would not know how to apply this to return v new visitors.

  4. Great post, Holly. Good advice.

    Experiments in progress often reverse themselves and swap places, which is why we need to be cautious about interpreting results and ensure you understand the statistical validity of the results.

    It’s statistical clumping that can be misleading if you try to analyze results before you have statistical significance.

  5. Interestingly now the results have reversed, from the alternative being up to 25% worse on conversions to being 11% better!

  6. Matt – I’m interested to hear what your conversion event is, and how many conversions you have so far in this test. Hopefully you’re using a tool that provides the statistical significance of the results for you in the reporting. Feel free to email me at lchiu[at]omniture[dot]com if you want to chat more!

  7. Well i have let the experiment run fr a while – traffic is not huge so it needed time. It seems that things have settled down to a 15-19% improvement between control and test.

    If one thing learnt – let the experiment run.

    Now the conversion even is those people presented with the A or B page and those who then make it as far as the page where they are presented with the order options.

    Basically the test is about providing a page that has enough information to encourage people to click to view the various artists and then go that next step to the page to select the caricature ‘Products’

    I have just implemented a funnel to track from there.

    So while there is success in getting more people to the select product page, sales have remained constant, so not sure if this means success or not.

    I am also experimenting with different pages for the selection of products (making it simpler and simpler but still clear in what to do), but my cart does not allow insertion of code to track which is best, so it is more a ‘feel’ exercise!

  8. great advice Holly

  9. It is also important to note that there has to be significant traffic before testing becomes useful as more data produces statistically significant results.

  10. Thanks Holly for this outstanding post. This is very true. You can’t tell if you are doing right without testing, testing and more testing.

  11. Just too late to see this post, great!

Add Your Comments


Print this Article

Holly Buchanan is a marketing to women consultant specializing in marketing to women online. You can read her blog at She is the co-author, along with Michele Miller of The Soccer Mom Myth - Today's Female Consumer - Who She Really Is, Why She Really Buys.

More articles from Holly Buchanan

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: