Comments on: Video: How to Do A/B Split-Testing on Lower Traffic Sites

By: John Quarto-vonTivadar

John Quarto-vonTivadar — Wed, 01 Jul 2009 17:51:49 +0000

these are two great issues to bring up! I would add that:

1) low traffic sites do have the issue of sufficient traffic but that is far less than is commonly understood, esp for people who are testing with a tool like GWO that (unfortuantely) distributes the test evenly across all variations. There are well-known techniques that allow statistically good testing with far less traffic than, say, GWO, requires

2) the fact that testing can be deterimental is certainly an issue but if no testing is being done then the only rationale prediction for the result of testing is “no change” — if you actually “knew” that a change would increase conversion (or whatever goal you measure success by) then you would simply implement. And if you “knew” the effect would be bad, you wouldn’t test at all, you’d jsut discard that idea. So one actually doesn’t know if improvement will occur or not, which suggests that a sub-set of traffic should be exposed to traffic (see point 1)

in addition, testing “blind” is the biggest cause of “deterimental” testing — what you really should be doing is working with an expert who knows what tests to do, their relative importance, and can help you establish for yourself the opportunity cost of NOT testing.

By: SEO Tips

SEO Tips — Wed, 01 Jul 2009 17:27:12 +0000

The main problem with low traffic websites are that they do not have much power to experiment or sometimes it an prove detrimental.

By: Property Doc

Property Doc — Mon, 29 Dec 2008 00:25:15 +0000

It’s important to remember that small traffic sites may not have a vast supply of customers to test with anyways. If we have a proven message that brings customers in, and if that’s only 1 customer a day for instance, testing a new strategy might lose that 1 customer for us.

Still waiting for the perfect win-win.

By: Soeren Sprogoe

Soeren Sprogoe — Thu, 10 Jul 2008 09:55:25 +0000

Also, don’t forget that if a test runs for too long, the result might be affected by seasonal fluctuations in your specific business.

Fx. running a test on a skiing trip landing page from November to February is bad, because the visitors are in totally different places in their buying cycle when comparing November to February, and thus they will produce different behavious and results.

By: Dr. Pete

Dr. Pete — Wed, 09 Jul 2008 21:16:28 +0000

Always nice to see Dr. Wilson. I’d say this is good advice even for 100 conversions/week. When a test run is going to take a minimum of 3-4 weeks, you can’t afford to test things at random, mess with low-impact changes, or try to run a huge, multivariate test.

One big problem I have on low-traffic sites is the increased temptation to check results too early. As a research psychologist by training, I ought to know better, but I still find myself peeking in the first week and wanting to stop the test when my favorite version is ahead. It’s a bad, bad habit.