Questions? (877) 643-7244
FutureNow Article
Friday, Aug. 1, 2008

A Guide to Common Testing Challenges

By Bryan Eisenberg
August 1st, 2008

To survive, companies must commit to optimization and testing.

It won’t be easy. Being good at something never is.

A few weeks ago I gave a keynote at the online merchandising workshop and had an opportunity to chat with several online retailers of all sizes. Each shared the challenges it faces as it tries to adopt a culture of optimization.

Last week, I shared with you what I learned from Raul Vazquez, CEO of from his keynote at the conference. Unlike many retailers, has a culture of optimization and embraces a process for continuous improvement. Even with its process in place, however, it doesn’t do any A/B or multivariate testing.

An executive from another multibillion dollar online retailer told me an all-too common tale of how it began testing with the help of a vendor. After seeing no lift and investing all the resources, the retailer cut the vendor loose and let testing fall by the wayside. Like so many others in these tight times, it axed testing as a budget item. Can anyone afford to stop optimizing?

The fact that optimization and testing challenges are rampant shouldn’t surprise anyone. In its “Guide to Retail and Web Site Design,” Internet Retailer’s research unwrapped the fact that over 76 percent of online retailers don’t test.

It’s not that online markets have a violent opposition to testing. It’s a combination of things. Many don’t know where to start; others may have started and had limited or no success. Here are some other common challenges that online retailers and online marketers face:

  • Simple slice-and-replace testing/optimization. This is the process of Web teams slicing and replacing an element, a page, or a portion of the visitor experience and finding little or no lift in the numbers. So they move on and try another. This is a challenge because this method has no way of determining if the original hypothesis was incorrect (the reason they made the change) or if it was an execution issues.
  • Resources. Testing well is hard work and uses resources: time, effort, and cash. And because teams are unsure, they resist or don’t give their best effort.
  • Platforms. Some site platforms make it difficult to test. Ultimately, you must determine what costs the company less: lost opportunities, platform upgrades, or changes to allow for easier and more efficient testing.
  • Lack of a process. Most retailers test randomly, just selecting something in the conversion funnel or their least favorite page. Others perform large-scale tests and may break or fix many things along the way. In both cases, they have no method for gleaning insight so they can successfully repeat the process.
  • Persuasion or brand perception issues. Often time the issues that keep visitors from converting aren’t as simple as a button color, an element’s position, or a form improvement. We recently worked with a big-name financial institution that wasn’t committed to taking the time to test these subtle issues. Instead, they went with another vendor that tested more simplistic conversion issues. Needless to say those efforts didn’t move the needle.

Most of these challenges are related to companies not setting a solid (i.e., more scientific) testing process in place. None of these challenges is a reason to give in.

If you have testing challenges, you should know two things before you throw your hands up. First, you aren’t alone. Second, don’t stop testing — instead stop testing badly by adopting a cycle of optimization and a smart process for doing so.

I will be launching (and signing) my new book, “Always Be Testing,” at Search Engine Strategies San Jose on August 18. The book is based on my 10-plus years of experience helping companies that struggle with testing and giving them practical guidance and tools to deal with these challenges. If you plan on attending, let me know. I’d love to meet up with you and talk about your challenges.

*Cross-posted on ClickZ. 

Add Your Comments

Comments (3)

  1. Bryan, fantastic insights as always.

    The core concepts behind testing/conversion optimization are exciting and empowering. Taking control of growing the goal outcomes of your website can be a heady brew for marketers who are new to this kind of work, which can lead to a rush to ‘boil the ocean’ by applying lots of creative ideas without a proper framework in place.

    With that said, I couldn’t agree more with your emphasis on starting with process. New practitioners of conversion optimization should take their initial excitement and energy, and pour it into building a solid plan to move forward with testing. The visible changes to the look and feel of your site will be smaller at first, but the visible results will be transparent. If changing the color of a button from blue to red grows your sales but you can’t prove it, your first cycle of optimization might be your last.

    Looking forward to reading more on process when the book comes out.



  2. Thanks for the link to the book. I normally don’t like Amazon’s practices even as effective as they are, but for books they usually are the best resource if I can’t get it locally.

  3. in such a way of IA was remotely correct, interested in easier. Grouping this is a cheap way to try to circular argument. IA in your build up each of the definition is a definition is never meant different personas

Add Your Comments


Print this Article

Bryan Eisenberg, founder of FutureNow, is a professional marketing speaker and the co-author of New York Times and Wall Street Journal bestselling books Call to Action and Waiting For Your Cat to Bark and Always Be Testing. You can friend him on Facebook or Twitter.

More articles from Bryan Eisenberg

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: