Website optimizer calculates probabilities of one combination outperforming another. When an experiment has a combination performing very well, in your example 100% better, it is much easier to predict that it will outperform the original version. When they are returning very similar results, say a 5% difference, it takes Optimizer a lot more time to see which one will likely win with any level of certainty.

I suggest reading the technical explantions of Optimizer, and particularly the section titled “All About Statistics.”

Test Combinations: 2
Page Views Per Day: 50
% Visitors in Experiment: 100
% Current conversion rate: 1
% Expected improvement:5

It gives me Duration: 10636.16 days.

But if I increase the % Expected Improvement to 100% , it gives me 37.6 days .

How is this possible? I was expecting more days for experiment if % improvement is more. While this calculator is reducing the number of days with % increase in Improvement.

Do you hate your Content Management System? Do you even have a content management system? Take a look at this to see what’s wrong with them.
……

Tom Leung, the Business Product Manager of Google Website Optimizer mentions in a series of webinars says Google recommends waiting until there have been 100 successes for a variation of a test to give it statistical relevance. They also recommend waiting two weeks so daily fluctuations will level out.

In addition to waiting for the necessary traffic to be exposed to the test, you could buy PPC traffic, but you must be careful that the words you target on the campaign are relevant to the landing page you link the PPC traffic to and are testing. If there is a disconnect, or loss of scent, from the PPC link to the landing page, your PPC traffic will be more likely to leave your site, and skew your test results.

P.S. Rafael,Did you run your friends through an A/B test, or show each of your friends you sent there two options and asked them to choose which one they preferred? I don’t believe that sending your friends through an A/B test would yield valid results, but asking their collective opinion is useful to know, but remember your friends may not be your customers, and it’s your customers preference you want to discover.

Also, I really liked the case study about your book in the New Rules of Marketing. It’s rare to find people sharing strategy in that kind of detail. Thanks for your transparency.

I think SEO, Usability, and Analytics are FOUNDATIONAL concepts, not after-thought concepts. The trouble is getting businesses and agencies to believe it.

Much of my professional day is spent in this conflict!

Thank you for sharing your situation.

If you are limited in coding the tests on pages with a testing program like Google Website Optimizer, you may have to swap out different versions of a page manually and track the analytics data. It is a bit more work, but definitely worth the effort to your customers and your business in the long run. An additional benefit to this route is when you are testing larger design elements, that would be very noticeable like color or the number of pages in the scenario, you can provide a link to the original version for any visitors that may feel their security is threatened by the new design.

Using a message to warn visitors they are going to see a “beta” version of your site is a tempting idea, but may work against you. If you are able to pick important elements and test them individually, you will gather valuable information before implementing a large redesign, which could possibly scare your visitors.

The ultimate truth with testing is that you don’t know until you test it. Make some hypotheses and find out what works best for your customers.

Thanks again for your comment.

If the users noticed a discrepancy between two different versions, they’d be very likely to freak out, thinking that they are a phishing victim.

How would you recommend to solve this? One idea that we had was to notify certain users that they will see a “beta” version of the website the next time they log in.

The problem is that this can only be done for pages that require an authentication. Which is not very effective.

Any other suggestions?