Comments on: Give Yourself a Winning Chance with Testing

By: canadian bodybuilding

canadian bodybuilding — Tue, 07 Apr 2009 15:04:25 +0000

great advice Holly

By: Caricature King

Caricature King — Tue, 17 Jun 2008 01:19:50 +0000

Well i have let the experiment run fr a while – traffic is not huge so it needed time. It seems that things have settled down to a 15-19% improvement between control and test.

If one thing learnt – let the experiment run.

Now the conversion even is those people presented with the A or B page and those who then make it as far as the page where they are presented with the order options.

Basically the test is about providing a page that has enough information to encourage people to click to view the various artists and then go that next step to the page to select the caricature ‘Products’

I have just implemented a funnel to track from there.

So while there is success in getting more people to the select product page, sales have remained constant, so not sure if this means success or not.

I am also experimenting with different pages for the selection of products (making it simpler and simpler but still clear in what to do), but my cart does not allow insertion of code to track which is best, so it is more a ‘feel’ exercise!

By: Lily Chiu

Lily Chiu — Tue, 10 Jun 2008 04:39:10 +0000

Matt – I’m interested to hear what your conversion event is, and how many conversions you have so far in this test. Hopefully you’re using a tool that provides the statistical significance of the results for you in the reporting. Feel free to email me at lchiu[at]omniture[dot]com if you want to chat more!

By: Matt

Matt — Tue, 10 Jun 2008 00:32:39 +0000

Interestingly now the results have reversed, from the alternative being up to 25% worse on conversions to being 11% better!

By: Chris Goward

Chris Goward — Wed, 04 Jun 2008 17:09:10 +0000

Great post, Holly. Good advice.

@Matt
Experiments in progress often reverse themselves and swap places, which is why we need to be cautious about interpreting results and ensure you understand the statistical validity of the results.

It’s statistical clumping that can be misleading if you try to analyze results before you have statistical significance.

By: Matt - the Caricature King

Matt - the Caricature King — Tue, 03 Jun 2008 23:51:31 +0000

“you might want to see if version B is the best performer for new visitors and version A is the best for return visitors.”

gulp!!

just feeling my way with straight A/B, – would not know how to apply this to return v new visitors.

By: Lily Chiu

Lily Chiu — Tue, 03 Jun 2008 22:05:31 +0000

Holly – glad you found my post useful! Your additional points are great. I agree that tests that don’t provide lift should not be seen as failures. The definition of “success” in a test should take both the lift and learnings into consideration. For example, if you learn that having models in your product images doesn’t affect conversion or revenue, that’s a huge take-away that allows you to skip on paying for models in the future! Also, imagine what happens if you don’t test and go solely on gut. The probability of pushing a “loser” out into production is probably pretty likely.

Matt – you make a great point about how important it is to allow your data to normalize over time. Depending on traffic and environmental factors such as time of day and day of week, I’ve seen results change pretty dramatically. In your example, you might want to see if version B is the best performer for new visitors and version A is the best for return visitors.

By: Matt - the Caricature King

Matt - the Caricature King — Tue, 03 Jun 2008 00:30:26 +0000

One issue I have with split testing is the way results can almost reverse halfway through!

In one test, version B was getting a 25% conversion rate, while version A had 0%

Two days later, Version A has all but caught up, so the different between the two is barely discernible.

I guess the only think to do is give it more time as environmental factors could influence it??