Questions? (877) 643-7244
FutureNow Article
Friday, Feb. 22, 2008

Website Optimization Starts With a Hypothesis

By Ronald Patiro
February 22nd, 2008


That may or may not be true. At the moment, this statement is merely a guess, an assumption — but it’s testable. It’s a hypothesis.

People love to insist that your website is made of magical ones and zeros. “It’s HTML,” they’ll say. “It lives on triple-redundant co-located servers,” they’ll argue. Yet the truth is much simpler, and scarier, than that.

Your website is a tower of assumptions. Everyone’s is. Perhaps yours was built according to a specific blueprint. Maybe it was built from a template. Either way, if it’s not properly maintained, the structure will collapse. But before you demolish the current structure and start over from the ground up, you should test the existing site.

If you want to improve your website, testing provides the scaffolding to fix it. And just as you wouldn’t hire a renovation crew that uses scaffolding made of toothpicks, your optimization tests require strong hypotheses. Of course, you can always test a bunch of random variables and see which configuration works best with your visitors, but that generally takes too long, adds noise to the data, and makes it difficult to gain any real insight.

The better thing to do is to start with a hypothesis.

Dropping Science

In my last post, I showed how testing allows you to optimize by letting visitors design your site for you. By giving them new versions of navigation and content elements and closely monitoring to see which ones work best, your visitors can vote with their clicks, and you can more easily adjust your site to fit their needs.

Be careful, though. If you don’t have a solid hypothesis, improvements can take longer — and be more incremental — than they should be. Recycling random variations of a page just to see what works often yields a much smaller return on investment than hiring a website optimization firm.

It’s the most common problem we see among companies that don’t outsource their testing: They don’t really know what to test.

Regardless of who tests your website, the scientific method [define] must drive the process. Your venture into testing must begin with curiosity. Curiosity is fundamental to humanity, and the basis for our achievements. To have success online, you must be curious as to why things happen and what is influencing them.

• Observation: “Why do so few people add an item to their cart from the product page?”

Observation: “Why do my blog posts with short titles seem to get more comments?”

Curiosity is the initial spark to start a learning experience, but ideas and explanations must be conjured to satisfy that curiosity. This is where the hypothesis comes from.

Don’t Believe the Hypothesis

Again, a hypothesis is just an assumption. The ideas and explanations you base this assumption on can come from real world examples or basic intuition. To write a hypothesis, simply take the action you’re considering and state the result — a benefit, we hope — that you expect it to have.

Hypothesis: “Making the ‘add to cart’ button larger will increase our conversion rate.”

Hypothesis: “Using blog post titles with six words or less will increase the amount of comments.”

The one and only purpose of running a website optimization test is to prove (or disprove) your hypothesis by exposing it to real world conditions. As such, you’ll need to create variations of the elements you wish to test in a way that properly reflects your hypothesis, so you can test them against the original version to see which one works best.

Let’s start with “Making the add to cart buttons larger will increase our conversion rate.” To test this hypothesis, you’ll need to create a version of the page with a larger add to cart button. To be sure, you may also want to test more than one size. If a large button isn’t ideal, maybe a medium-sized one is.

Wash, Rinse, Repeat… TEST

Lets say the test proves our hypothesis to be valid and you decide to make the “add to cart” button larger. Wonderful, but you might want to hold off on the champagne.

Now it’s time to create another hypothesis about the best color for the “add to cart” button. For instance, “A green ‘add to cart’ button will yield a higher conversion rate than similar red or blue buttons.”

The point is to learn something — anything — about what is and isn’t working on your site. Approach testing in a systematic way and record what you learn to guide you through future tests. You may also want to revisit certain tests to see if they still hold true, especially if you’ve changed other elements on the page.

It’s very important not to get discouraged. Even if your hypothesis is disproved, you’ve learned something valuable; that what you have is working well enough for you to focus on another area of your site that needs attention.

On the other hand, if your hypothesis is strong — and the test results prove it — you’ve begun remodeling your “tower built on assumptions” into a high-rise casino, where the odds are stacked neatly in your favor.

. . .

[Editor's Note: Blinded by science? Need a renovation? Future Now can help you test it.]

Add Your Comments

Comments (20)

  1. Thanks for the great advice. Very helpful.

  2. [...] tagged randomOwn a WordPress blog? Make monetization easier with the WP Affiliate Pro plugin. Website Optimization Starts With a Hypothesis saved by 9 others     shugotenshineko bookmarked on 02/24/08 | [...]

  3. It’s today I was thinking the same thought. Will use it to make my new site.

  4. We are in the process of a site update right now, and this is very timely. Outsourcing and building testing into the reconstruction seems the way to go. However there are a large number of variables that could (and should) be tested; the issue is finding (and testing) those likely to give the highest returns. Smaller companies often do not have the resources for full-scale testing so determining what these might be is imperative.

    Your point is extemely valid: “Recycling random variations of a page just to see what works often yields a much smaller return on investment than hiring a website optimization firm”.

  5. Great points. One of the things that’s really starting to concern me about the way that people use testing is how we treat it like magic. Pick a Version A and Version B, test it, and do whatever the test outcome says. The problem with that approach is, even if that were always reliable (and data does lie on occasion), you don’t learn anything that you can use in other situations.

    Having a hypothesis means you drive testing, thoughtfully and purposefully, instead of letting testing drive you.

  6. Amen Dr. Pete!
    After a very recent experience – I think the most important sentence in this blog:
    It’s the most common problem we see among companies that don’t outsource their testing: They don’t really know what to test.

    Last year my web folks sold us a bill of goods – testing multiple scenarios of pages, click thru, etc. And in the end – all the results were totally inconclusive. Now my personal cynicism aside about corporate politics, procedures, (and all other excuses I can think of) – Please learn from my mistake…
    Get involved from the idea conception – don’t leave it up to just the designers. Not knowing what to test proved to be a big waste of internal resources and $$$’s. For me it was like ‘letting fox watch the chickens’ – Sure it looked good on their dept’s yr end objectives, but it impacted my product ZERO. (sorry – had to vent)

  7. [...] ugly, but what Bezos realized early on is that, to be a successful online merchant, you need to get a hypothesis and test it if you want something that [...]

  8. [...] test. Without a clear hypothesis, it's easy to get hung up on your own guesswork. Knowing how to get a hypothesis is critical. So, by popular demand after many Website Optimizer users asked for more insight on [...]

  9. Just a quick note since many of you will be interested: We’re teaming up with Google for a free webinar on Tuesday March 11th on how to get the most out of Google Website Optimizer.

    Hope you can make it!

  10. That’s good advice. One thing designers forget that sometimes feelings about new tools change all the time. Something can seem great at first and then later on people realize it’s not so great. Or what works for some may not in fact be working for a lot of other people. Being able to constantly check for information about what is and isn’t working is always a good thing.

  11. [...] from people who want to know the secret. They've read a few posts on website optimization testing, but they're disillusioned. Their tests haven't been effective — meanwhile, Future [...]

  12. [...] Know what you’re looking for — Make sure you know how to get a hypothesis worth testing. In other words, you should know ahead of time how to interpret the results. [...]

  13. [...] will serve well as a starting point. For more details about a good hypothesis take a look at “Website Optimization Starts With a Hypothesis” over at [...]

  14. great article.

  15. I am just getting our website updated at the moment and this post has given me some great ideas to approach the web designers with. Thanks for the information.

  16. I agree 100% with you. I think that the best thing is to test it and see what works the best, and try to ignore all hypothesis.

    Some hypothesis will work on you web, some won’t.

    Smart webmasters will test which one works, dumb will just read everything on net and use them without even thinking if this is true.

  17. Great text, I suspect that something sort of like EMH dominates the find engine world, but instead of prices for commodities at points in time, the market is search engine rankings and keywords. In my inchoate model, each keyword is its own market. The “price” of a given website would be how well it ranks in a search for that keyword.

  18. Great informative article – some comments if I may :-

    1. I’d say most people here use an off the shelf ecommerce package, it’s going to be rather hard to duplicate products as it’ll confuse customers, also Google will mark you down for duplicate content. Change your content – you invalidate your test.

    2. If you try and follow nature – we either innovate or we immitate. Again, I assume most dont have the resources to run all these tests, so the natural instinct is to copy the top dog (just like we do in fashion & music etc) , and then make our niche. Maybe I’ll try and look like amazon or iTunes ?, just like the Beatles immitated, the beach boys, Elvis etc and every pub band then plays the songs that are in the top 40.

    It also depends on your market type – eg 20 wholsesale customers can order the same total value as 80 retail customers. Those 20% wholsale customers can be converted via a phone call or email, their decision based on price and service – rather than the checkout button size and colour
    But – that is just my hypothesis :) – and probably wrong

  19. Sometimes we overcomplicate things. Especially building a website.

    When I build one, I put myself into my visitors shoes and ask myself, “What stands out best?” “What attracts my eye so that I want to stay and read more about this?” “Can I get what I came here for?”

    I believe the whole site browsing should be easy, fun to use, attractive to the eye, and load quickly.

    “Keep It Simple Smarty” is my motto I follow. I treat every visitor as an intelligent person trying to find what they need.

    Mix it up and make the site look pleasant. Making the site as interesting as possible without overdoing it is the goal.

  20. [...] Website Optimization Starts With a Hypothesis – Ron hits us up with an awesome “must read” follow up post to his previous post titled “Let Visitors Design Your Site For You”. [...]

Add Your Comments


Print this Article

Two legs, two arms, and 10 typing fingers!

More articles from Ronald Patiro

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: