Questions? (877) 643-7244
FutureNow Post
Tuesday, May. 4, 2010 at 9:07 am

Testing Woes 101: Lessons From a Messy Test

By Melissa Burdon
May 4th, 2010

Anyone ever done testing on their website undoubtedly has run some poorly-designed experiments; that’s how we learn.  It’s easy to make subtle mistakes in the design of a test that cause “messy” results, but read on to learn how to avoid a few of them.

FitnessAnywhere offers a product called the TRX that is a very effective portable gym.  One of their highly trafficked pages is the “personal fitness” page.  Once we got into the heads of their target customer via personas, and understood what their intent and motivations are when they arrive at this page, we quickly identified the things they should be testing.

personal fitnees- FitnessAnywhere-originalscreenshotThis is the original version of this page [click to enlarge].  We wondered if the image of a man working out was resonating with the women that were coming to the site.  We also hypothesized that the “watch video” call to action wasn’t standing out effectively in the active window due to its placement, and we knew how persuasive the video is in regards to helping the visitor truly understand the product benefits.

The other thing we noticed was that there weren’t any text links to help visitors find answers to the various questions that they might have while reading the content on this page.

Finally, the benefits that this product offers visitors were hidden all the way at the bottom of the page.  This was incredibly valuable content that we knew would work more effectively if it were brought further up on the page.

Should I ALWAYS Test One Thing At a Time?

FitnessAnywhere started to move forward with implementing the recommendations we gave them to fix some of these challenges.  Ideally, they would have implemented each change one at a time, and tested them separately, but then this blog post never would have been written.  ;)

The downside to testing one thing at a time is, if you get very little traffic, or the changes are so slight, you may have to run a single test for a very long time in order to see a test result. The benefit of testing one thing at a time is that you will know exactly what caused a lift or drop in the results.

If you test several things at once, all you will see is the aggregate result of those changes.  This means, if the aggregate result is negative, it doesn’t necessarily mean all of the changes were bad – just that the ones that were outweighed the positive effect that the others had…. or vice-versa.  In fact, it is far more likely that some of the changes are good and some are bad.  You don’t know which changes caused the overall change, and which ones tempered it.  It’s even possible that what looks like a negative change is actually a positive change in disguise (if even one of the changes is having a positive impact that is being dragged down by the other changes).  There’s a reason why your science teacher taught you to “isolate the variables” in your experiments!

personalfitness-fitnessanywhere-variationThese were the changes FitnessAnywhere made.  Click to enlarge.

WHAT success point should I test?

Sometimes it’s obvious what the conversion goal of a test should be.  Other times, it’s not that obvious. If you’re testing your homepage or an informational/category page, you might be stumped about what to tag as your conversion goal.  Sometimes, you might feel that you should test various conversion goals separately, and run a few tests with the same variations, using different goals to see what the results actually are.

What you need to do is identify your business goals, first and foremost. If you think that the changes you’re making will potentially result in more traffic moving to a product page, tag the product page as the goal.

FitnessAnywhere identified their video as a micro-conversion point that they wanted to track as the goal for their first test on this page.  Since they were making more than one change at a time, there wasn’t a clean, single goal we could isolate.  We felt, however, that these changes would help drive more traffic to watch the video, and that watching the video was a good thing.

Website Optimizer - Personal Fitness Page TestThese were the results of their test [click to enlarge].  What was nice about this test was that we only had to run the test for a few days to get a strong enough confidence level to complete the test. This resulted in an observed 201% improvement of the traffic who was clicking to watch the video on this personal fitness page.

One problem with this slightly messy test is that we aren’t sure which of these changes resulted in more visitors watching the video.  We can make some pretty good educated guesses, but again, because we made so many changes at once, we don’t have proof.  What if one of these changes had a slightly negative impact on the results and this improvement rate could have been even higher?  Results would have been more conclusive if the client had simply made one change at a time, and run these as separate tests.

Another challenge is that it doesn’t tell us whether this improves overall sales.  Does the lift in video-watching result in more products being added to their cart?  More people checking out?  If analytics were set up to collect both the rate of visitors who watch the video and end up adding an item to their cart, we could run the numbers based on this single test and feel confident about how this win results in overall sales.  Since that data is not currently available, we aren’t sure yet, even though our hypothesis at this point would be “yes.”  To get around that, we have recommended that the client re-run this same test with the conversion point being the shopping cart page.

We can still learn from a messy test, and we can probably learn even more by cleaning up a test or changing parts of the same test and running it again.

Add Your Comments

Comments (55)

  1. Yep doing one a time is defenetely more good, but also depends of the time that u have.

  2. To understand what the customer wants, we should be put in its place. Above all, we should be familiar with the product very well to describe his best qualities.

  3. [...] Testing Woes 101: Lessons From a Messy Test, GrokDotCom [...]

  4. I think it is important to always be testing a variety of efforts. There are many things you can do to landing pages and websites and if you don’t take the time to really improve you could be missing out.

  5. Thank you for this post. I understood better why we have to at least change things little by little so that at least we can gather accurate data of what works and what does not.

  6. Anytime I change something, I always make a big change and then mess up everything on the page. A lot can be said for little changes.

  7. I can say that the final look of the page is waay better than the old one. The woman in the header made the site more appealing to its readers (which according to the article are women). The picture somehow relates to the readers.

  8. I’ve made that mistake on my website, changed a few things in my Yahoo campaign and a couple things on my website, now conversions are down:(

  9. It is always better to do change one at a time so we can track the result perfectly. However if we do not have time, let’s do all of it in a time.

  10. It is hard to isolate what changes create the right or wrong result so one atbantime seems best

  11. Thanks for the useful lesson. Now I make changes little by little in order to receive the data regarding success and losses.

  12. Thanks for this post on testing. Testing is definitely something that I need to start focusing more on and you give some great tips on how I can do so!

  13. Thanks for the useful lesson. Now I make changes little by little in order to receive the data regarding success and losses.

  14. #

    I’ve made that mistake on my website, changed a few things in my Yahoo campaign and a couple things on my website, now conversions are down:(

  15. Pssst…Melissa–first sentence… I think you’re missing a couple words there along the lines of “who has” … :) (Great article!!)

  16. thank you vert much

  17. Ya. You can really hurt your track if you tweak something you think is insignificant which turns out to be something major.

  18. thanks for your report

  19. Interesting article and analysis. It’s far past my limited experience though – I’m fairly new to web design, and with my “artistic” sites I just sort of wing it and go with what I think looks good.

    Other sites I tend to go with a more generic basic format – depends on what I’m aiming at.

  20. Nice, it is better and better!
    Thanks for this post on testing. Testing is definitely something that I need to start focusing more on and you give some great tips on how I can do so!

  21. Thank you for the informative post. I personally find that taking my time and perfecting one element at a time saves me headaches later.

  22. Testing is always risky. When changing many page elements at once, it is very hard to determine which changes affect the traffic to the site, so slowly changing one element at a time is our preference.

  23. Doing one test at a time is ideal if it is a ‘true test’ and you are looking to find out if something is better than what’s in place already. However, if you know changes need to be made (as mentioned in the article) then it’s more of an improvement than a test. In this situation, why wait? Just go for it.

  24. You certainly can test more than one thing at a time, just be sure that you are tracking it properly. The results you get will tell you which page converts better leading to a better ROI. That can make the difference between a successful and a failed campaign.

  25. It is best to test one thing at a time in my experiences, i also like to get others to road test my site and comment and how easy it is to use etc

  26. I think an important point has been missed. Whether it was one thing or another, you did increase their traffic by 200%. That’s positive nonetheless. At this point you can reverse one change at a time and see what the outcome is, without a likely risk of losing 100%.

  27. I think the bottom line is that when you have very little or no traffic, it is always going to be a hard to make definitive judgments. All in all I think your optimization was pretty impressive :)

  28. I am actually interested in seeing the results of a finely tuned test as compared to the results from this messy test. I wonder if there would be a significant difference.

  29. I realized that testing the market is important with any business. I made the mistake of going into a niche without doing the research. Great post!

  30. Thanks for this post. Very interesting. Testing is definitely something that I need to start focusing more on and you give some great

  31. i think that these kind of testing are for the braves

  32. With more and more people using domains to bring traffic, it is important to address exactly what they are looking for on the landing page. Really helped decrease my bounce rate across other sites.

  33. Takes quite some time to do these tests for multiple websites. The results are clearly getting better and better though!

  34. Ok you say that testing one thing at a time takes to long if there is not much traffic. But if I am not getting a lot of traffic isn’t the whole testing like totally pointless? Shouldn’t I wait with testing a lot until I am getting some decent traffic so that I can see results?

    Just my thoughts. I always try to change a small thing see how that works out and then test another.

  35. My tendency is to shake things up when I don’t see results. That isn’t always the right way. Small changes, monitor those changes, and be patient.

  36. Anytime I change something, I always make a big change and then mess up everything on the page. A lot can be said for little changes.

  37. Sometimes I have the idea that tests are easy to sell. Some account managers say ‘lets test this or that’ and the account immediately agrees. However, it may be a bit more smart to argue a test with the marketing department or so.

    Keep up te good postings!

  38. great post!

    Must change many times , to get better!

  39. I had never thought of website changes in this way. In the past, I’ve been a get it all done at once sort of guy, but now I will be rethinking that approach.

  40. I once heard that the only thing constant in life is change and it has been so very true. I must adapt to the moment in order to be all that I can.

  41. I think changes should be made in phases and everyone should avoiding changing everything in one time. I did that and after changing whole design I had to change it back cos my conversion went down. Afterwards Ive made in it 4 phases and thanks to it I could noticed what I was doing right and what I was doing wrong.

  42. i’m curious…why the effect named “messy”, just like argentina’s football player…..right?

  43. Film

    My tendency is to shake things up when I don’t see results. That isn’t always the right way. Small changes, monitor those changes, and be patient.

  44. Well, we definitely learn more from a great big mess than from a squeaky clean test. Thanks for the informative post!

  45. thanks for your report

  46. thanks for all people really so nice here

  47. Great Report, Keep up the good postings!

  48. Thanks for this post on testing…If you are looking for PMP mock test, SCJP 6 mock test, SAT practice test than contact me immediately

  49. Great insight, we’ll keep this in mind for our own site redevelopment. cheers.

  50. Is there such things as lessons from a messy test? I think you make some good points, but aren’t you going for consistency above all else?

  51. Nice lesson,it is very important to test one thing at a tie in other to get the best and accurate result for the test. Moreover, anything that can benefits visitors or customers should always be place on the top of the web page, for easy access, because not everyone will read all text on a web page.

  52. Good lesson. I suppose it’s a bit like the children’s story about the tortoise and the hare. If want consistent results it’s better to do it slowly, understand it and get proven results instead of quick fixes.

  53. [...] Testing Woes 101: Lessons From a Messy Test [...]

  54. [...] Testing Woes 101: Lessons From a Messy Test [...]

  55. When you test multiple things at once you really have a hard time narrowing down problems. so much online is interconnected, especially in conversion…

Add Your Comments

 

Print this Article
Share

Melissa is a Senior Persuasion Analyst at FutureNow.

More articles from Melissa Burdon

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: