Questions? (877) 643-7244
FutureNow Post
Thursday, Jan. 6, 2011 at 10:32 am

Use Testing to Make Business Strategy Decisions

By Natalie Hart
January 6th, 2011

We all know that testing is a great tool to use when deciding between images, content and layout, but have you ever considered it to make business decisions? Recently, a client came to me with a predicament: UPS is raising their shipping costs by 5%. My client wanted to know if it would be best to swallow these costs or pass these charges onto their customers. Obviously, they didn’t want to jeopardize their customers; however, incurring greater shipping costs was something they weren’t crazy about either. Fortunately, our groundbreaking OnTarget strategies can address problems such as this. So, how do we do it? Test it, of course!

Formulate Your Test Question (or Hypothesis)

I worked with them to create the following research question for the test:

With UPS increasing our rates, we will encounter a 5% loss if we swallow this cost. However, before making a decision, we want to test what our potential loss may be if we pass this loss onto the customer. If the potential loss is higher than the cost of swallowing this 5%, it would be wise for us to either take the 5% loss or seek out other shipping options.

Whenever you want to create a test, first think about why you want to test it. This is true for design tests, copywriting tests, content tests, headline tests, image tests, and business strategy test such as the one we are illustrating here.  In short: any test begins with a question. You should be able to formulate a hypothesis or at least a research question. If you can’t get this far, then it’s probably not the best test to run.

Valid Conclusions are Rooted in Valid Data

It’s impossible to draw conclusions from data you can’t trust. This means you not only need to know what each discrete piece of data you are collecting represents, but also be able to ensure that each piece of data is clean. Diligent collection and tracking of data is an essential component of any test. This is when being anal is a plus. Any other tests being conducted on the site should be paused while this test is running. You also want to make sure you have solid, consistent data collected prior to running the test so you can look at not only your winning variation but also compare to past data to ensure you’re making the right choice.

It also wouldn’t be wise to run this particular test during a time where you’re experiencing a large influx of late stage traffic. This means that holiday seasons, or times when there’s an annual sale would not be the right time to test. Yes, I remember my recent post about why the holidays are the perfect time to test, but there are caveats to that, and this is one of them. Holidays and sale times push visitors toward the late stage of buying more quickly, a moment when they may be more heavily influenced by price, so tests that influence pricing should be put on hold for another time. Be sure to give the key issues involved in your test, and how other business factors could skew or be skewed by those issues, some critical thought before you decide when to run a test. Each test is unique, and comes with its own constraints that need to be considered and worked into the testing solution. It’s part of ensuring clean data.

Test Away!

Testing is a valuable tool that can be used in a variety of situations, so don’t forget it when you’re making higher level business decisions bigger than what color button works best (but test that too)! We’d love to hear how you used testing to inform your business strategy, so share your business strategy testing story now.

Add Your Comments

Comments (42)

  1. It is a good point that we should avoid to conduct this type of test during holiday season. It will definitely skew the result. By the way, I’m just curious what the result was…could you share the result?

  2. good business strategy is very difficult in practice, but if you truly study it, your business will run smoothly

  3. Yes I agree holiday seasons can give very string positives and strong negatives.

    However you need to be aware of what seasonality happens in different marketpalces at different times,a nd begs the question what is a usual month ?

  4. Completely agree with the point regarding the quality of the data, from my own experience checking the quality of the data before proceeding to do any analysis is paramount.

  5. Yes, testing proof of concepts or strategies minimises the risks!!

  6. There is no way today to get investments without thinking of ROI. So… testing is a rule!!!

    Nice post!

  7. A good business strategy is still difficult, and if we analyze the good and seek the best techniques, to be sure the work will always be good.

  8. I make a lot of IM strategy decisions via testing. I’ve experimented with product pricing and measured the conversion rate of pages in order to ascertain the best price point.
    One of the most famous tests of this kind, as far as I remember, was the one done by Amazon – they introduced free shipping ( I think it was in France first) and noticed an immediate surge in sales. People like the word free – and the sales data showed that including the shipping cost in the overall price, even though the ultimate prices added up to the same, caused an increase in sales.

  9. any test begins with a question! so true, and its so important to get that question right! Knowing why you need data and how it should be used is crucial and frequently people miss the point which limits the opportunities!

  10. How long do you run the test? Is a week or a month enough to come up with a valid decision?

  11. Enough sufficient data as well as sufficient time testing the data is important. Changing a keyword ppc strategy for example, is a big decision than can effect a large revenue stream dramatically. But the change in choosing a new keyword strategy should only be made after a sufficient time period has gone by for testing. This type of a decision should not be mad in haste for sure.

  12. Testing is everything. By changing a small thing you can really make an impact on your conversions.

  13. In order to solve every problem or making an excellent decisions we have to apply the basic approach in resolving a problem. During high school I’ve learned about the fundamental scientific method- and it really includes testing. To test the effectiveness of your business strategy helps improve the outcome from every decisions you make.

  14. I agree with Just_Search, “good business strategy is very difficult in practice”. True

  15. Valid Conclusions are Rooted in Valid Data – this is what most people do not remember as they use testing to make decisions.

    You cannot make a valid conclusion unless you have the data to back it up – regardless of what you think you know. Put time into validating your data, and it will pay off in the end.

  16. Interesting situation, but if I were to test it, I would be much intricate to do a split test on different products. Perhaps include the additional 5% payment for UPS on bigger products, while on the other hand absorbing the loss on costs for lighter products, I think that makes sense.

  17. Regarding the holiday season. I couldn’t help but notice in our data, that customers seem to care less about the price, the closer Christmas gets. This observation tempted us to slightly raise the pricing during the last week before Christmas and we did not see any decrease in conversions. The late buyer feels more pressure to get a present :)

  18. Although the quality of data is important, creating a perfect hypothesis is equally important. If the hypothesis is set incorrectly, the whole testing can go haywire.

  19. Great post – people don’t utilize testing enough where it matters. If you have what you think is a good new idea for your business, it is in your best interest to test it out on at least a subsection of your market – you have little to lose and a lot to gain, as long as you take quality data!

  20. @Jack of Tonerboss – that depends on the traffic level and number of variations. Check out Google’s handy duration calculator tool.

  21. You need to be really clear about the numbers you are comparing.

    Split A could be the lower price and you would compare the total revenue and the TOTAL GROSS PROFIT for that split with split B – the increased price.

    Split B – the increased price – could end up with fewer conversions and possibly lower total revenue but higher gross profit.

    The hypothesis at the start of the article in italics is confusing because 5% increase in shipping costs does not mean a 5% loss for your company.

  22. @Lingerie – Yes, good conversion rate optimization is a business application of scientific methodology: 1) observe data or nature 2) form an hypothesis 3) outline an experiment to test the hypothesis, specifying the constraints and circumstances under which it is true or false 4) run the experiment and observe what happens/collect data 5) evaluate resulting data 6) form conclusions and next steps 7) REPEAT.

  23. “Valid data” is the key here, because most people aren’t going to believe you. My favorite websites (other than this one of course) is those that involve A/B testing. You can make assumptions and guess all day long, but the numbers don’t lie. Thanks for another great post Natalie…

  24. i agree trial and error leads to the most knowledge, and efficiency. got to test things. however its not always so easy if its a big expense to test

  25. I think it was a few years ago when I started doing Adwords and learned all about A/B testing. That changed the whole way I do business from that day on both online and offline.
    But I guess that testing doesnt always lead you to the right place. I think that like you mentioned most of the mistakes involved in taking decisions based on testing is taking into consideration the wrong or lack data. That’s what I see as the problem with people around me at least.

  26. Indeed, testing your strategy will let you know which is better. You wont lost anything if you try it. If you fail, there is still a chance to recover. At least you tried to test the strategy. You test the strategy. You formulate the strategy and observe what happens. If it turns out good or bad, you will then know which is the best strategy for your business. Thanks for sharing an interesting post Natalie.

  27. Many business’s fall into the trap of making knee jerk decisions when faced with change in their business. I prescribe to the “wait and see” method. Sure your profitability may be affected within the short term, but if your business is sound, taking that hit initially may see you outlast your competitors resulting in more customers in the long run.

  28. how long the test ? if longer will not effective to business, need faster and take action soon

  29. Some of these things are just really hard to test it seems like…

    Experimenting though, and trial and error are key. I gotta hand it to you there.

  30. Many times we make the wrong strategies

  31. Doing business with good business strategies is really some what difficult,but it actually gives profit

  32. Oh ive made the wrong decision so often and it´s always good to be reminded to take a deep breath and focus …

  33. In today’s the most competitive scenario testing has become an important business tool to take proper and right direction. I took small decision on the basis of my past experience but for big and very sensitive decision is taken on the basis of testing. Testing needs little amount of time and external cost as well but I can bear it, to take right decision.

  34. Trial and error in business is like you are gambling also because you don’t know if the business will work.

    This post is very interesting and this one also

  35. @Ronald – you are right that she says some interesting things. It sounds as if English is not Judith’s first language (based on some of the phrasing she uses in her posts), and perhaps that is one reason for the only item I’d take issue with:

    Conversion rate optimization is not black magic, is a scientific method of analysis, which allows you to monitor different parts of the website to see what the response will generate more visitors to your website.

    Judith does go on in her next paragraph to say that you can increase conversions on your website, and to further explain why this is so important. However, I would just point out that the PRIMARY goal of CRO is to increase conversions… getting more traffic to your website can happen sometimes, but it is more a consequence of the conversion focus, rather than an outright goal. Otherwise, Judith is spot-on!

  36. An organized and systematic decision-making process usually leads to better decisions. Without a well-defined process, you risk making decisions that are based on insufficient information and analysis. Many variables affect the final impact of your decision. However, if you establish strong foundations for decision making, generate good alternatives, evaluate these alternatives rigorously, and then check your decision-making process, you will improve the quality of your decisions

  37. I own a small company and it is always better to pass on all costs to the customers. The costs in question are usually small and will not cause problems for the customer but will definitely add up for you.

  38. Testing is key – we test on regular basis, and either automate tests or schedule test periodically. We can’t depend on user feedback, which is reliable, but most users / customers are not patient enough to report, they simple move on. Tests are simulated user interactions with your company or site.

  39. Testing is really fundamental to most IM strategies. Unfortunately the problem we run into a lot, is the client’s unwillingness to “waste” time and money in a testing plan, when they just want to “get on with it” now. This makes it a difficult sell sometimes, however when we do it, the results speak for themselves, and when the results can be measured in pure revenue terms with a very simple ROI calculation, the rewards for the business are tangible

  40. Testing and getting valid data applies to so many business functions.

  41. I agree with some of those above that testing is fundamental and patience is the key to success.

  42. Learning from previous mistakes is always a positive move.

Add Your Comments


Print this Article

Natalie is a Persuasion Analyst with FutureNow.

More articles from Natalie Hart

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: