The ABCs of A/B Testing

My friend Cindy has taken up hiking, and, being a shoe kinda gal, she has taken an intense interest in hiking boots. She's the first one who will tell you, "They may be a dream on the shelf, but you really have no idea how they're going to work for you until you start trail testing them."

You could say the same thing about ebusiness. When you start doing business online, you enter the realm of "trail testing." And here is where you discover that the practical work of managing any online enterprise has only just begun. Because no matter how thoroughly you think you've covered your bases in the development phases of your Web site or your email campaigns, you are about to discover countless opportunities for improvement.

How do you take that first step in monitoring your performance based on all the little tweaks and fidgets you think will help? Ah, it's as simple as A, B ... and no C!

Maximizing your conversion rate is not simply a matter of making changes, it's about making

a) the right changes,

b) at the right time,

c) in the right sequence, and then

d) evaluating the results before continuing the process.

If you are not methodical in your approach to change, much of your effort will be wasted. So take your time, and keep these guidelines in mind as you test:

Always clarify in your mind what you are testing and how you are going to measure and interpret the results before you begin. You cannot measure success unless you know exactly what you are measuring (and naturally I hope you have a clear idea of what constitutes success).
Your test groups should be of similar size.
After the first test, you should always test against a control. The first time out, you are testing two unknowns, and you won't be able to determine the better option until the results are in. Once they are in, then you have a benchmark, or control, against which you can measure subsequent changes.
Remember you can accurately test only one element at a time. Even if they all seem necessary, changes need to be made individually so you can track effectively the result of the change. While you might institute several changes and see an improvement, it could be a "net" improvement - that is, a 5% improvement could be the result of change #1 helping 10% while change #2 actually hurt by 5%. If you make one change at a time and then discover it doesn't help, it's easier to back up and try something else.
If you are testing emails, send your test emails simultaneously to eliminate the timing variable. Ideally, at the same time, you want to test your Web site as well. Software platforms like Optimost and Inceptor can help with this.

A/B testing - sometimes it's called an A/B split - is the simplest and easiest method of testing elements in your emails or on your Web site. You divide your audience into two groups. You expose one group to the original version of whatever you are testing. You expose the other group to an alternative version, in which only one element has been changed. Then you track the results.

For example, suppose you want to figure out the best subject line for your promotional email. Prepare two separate emails, identical except for the subject line. The email with the first subject line goes to half your list, while the email with the second subject line goes to the other half.

To gauge the effectiveness of your subject line, compare the open rates between the two groups. Once you find your winner, it becomes your control or benchmark. Test it against another subject line, and so on, until you are convinced you've found the best possible subject line for your messages. And the results may surprise you. In one of our testing scenarios, one subject line generated an open rate 300% higher than its closest competitor. A difference like that can have a tremendous impact on your bottom line!

What should you be testing? It's easier to make a list of what you don't need to worry about, because that would be a blank sheet of paper. But here are some possibilities to get you started:

Emails: bonus gifts, coupons, P.S. messages, guarantees, opening sentence image, closing sentence image, from-field, calls to action, opening greetings, type styles, layout elements, graphic images, etc.

Web Sites: landing pages, language of copy (headings, body, calls to action, assurances), colors, location of elements, look/feel, hyperlinks, etc.

A/B testing is far from rocket science (and there are other more complicated and robust ways to test), but it has a sweet advantage: it isn't complicated. More importantly, it means you won't have to make potentially expensive decisions based on your gut reaction. You'll know, because you can say, "Here, look at the numbers!"

click here for a printable version of this entire article

Latest News: AOL is The #1 Converting Search Engine?

Our recently published study reflects more than 29 million search engine visits to dozens of e-commerce sites using WebSideStory’s outsourced HitBox services (www.hitbox.com). The e-commerce sites used in the study cover a broad range of products and services, everything from gifts and electronics to shoes and office equipment.

Why does AOL beat out the likes of MSN, Yahoo and Google? Our analysis answer why different search engines have such different conversion rates, the factors that may be impacting those conversion rates and what marketers can do to improve their results. Find our more about our exclusive study Converting Search Engine Traffic: A Q4 2003 Conversion Rate Marketing Report for Online Retailers.

Final Notice on Our Copywriting Contest!

You’ve heard me say it often enough, “Speak to the dog in the language of the dog about what matters to the heart of the dog.” Meat matters to the dog. That’s what gets the dog so excited he can’t wait to get his teeth into that meat.

For this contest, we’re asking you to write sales copy to promote that splendid book Persuasive Online Copywriting (yours truly on the cover, no less). But don’t write just any old copy. Write copy that will do the same thing that meat does for the dog. Identify the critical message, in language folks will find appealing, that gets them excited about sinking their teeth into this book.

You may submit as many entries as you like up until the deadline, November 17, 2003. Send them to TheGrok@grokdotcom.com. We’ll fly the winner from anywhere in the 48 states to New York City, expenses paid, for a day of in-depth copywriting training at the offices of Future Now.

Have you checked out the other places to meet us on our latest event schedule?

Keep ‘Em in the Cart

If you’ve got a shopping cart – or bag or basket or whatever – you’re probably aware of a little problem. Not everyone who enters your checkout process manages to complete it. That’s why, when I talk about the buying process, I make a distinction between deciding to buy and actually completing the purchase.

Between those two acts, your visitor must navigate the obstacle course of your shopping cart process, yet another contained conversion system (typically a third-party software solution) that requires multiple clicks and the willingness to part with some amount of personal information.

But, like I said. You are probably intimately aware of your shopping cart abandonment rates, because you monitor this sort of useful information. Right? Of course you do.

So today I’m going to ask you this question: Should you be turning your attention to refurbishing your shopping cart?

A Matter of Priorities

It’s an important question: before you do anything, decide if the next issue you need to address is shopping cart abandonment. Yes, you are losing folks there. Everyone does. There is no 100% conversion rate because this is not a perfect world. Even when you’ve covered every base imaginable, humans will still leave for reasons you can’t do anything about.

But if your Web site is typical, you don’t lose the majority of folks from your shopping cart. You lose them much earlier in the process – at the home page, or just a page or two in. If this is you, I’d say your efforts are better spent, for now, encouraging more folks to stay with your buying/selling process.

Because fewer folks bail out of your shopping cart compared to the number who bail out before they even get to checkout, improving your abandonment rates might take you from, let’s say, 3% to 3.3%. Sure, the difference is not to sneeze at. But keeping more visitors in your conversion process from the get go helps you increase your conversion rate by multiples.

The Metrics to Help You Decide

If you are on top of your Web analytics, you’ll know where your biggest problems are. These are the key metrics that will help you decide where you should be directing your energies.

Single Page Access. This is a percentage of the number of people who landed on a particular page – it might be your home page or it could just as easily be a landing page – and made it their first and last page. In other words, they came, they saw, they bailed.

You really want to consider these pages before anything else. What is wrong and how can you remedy it? I mean, we’re talking folks who weren’t even inspired to look a single page further into your site!

Scenario Analysis. Ideally, you want to be able to follow your visitor’s path through your Web site. Is he clicking where you want him to click? Is he following the breadcrumb trail you have so painstakingly crafted for him, only to jump ship on the fourth page?

If they don’t get to the shopping cart, you certainly don’t need to worry about them abandoning it. But let’s give them the chance. Look at the pages they reject and evaluate how to improve things.

Shopping Cart Abandonment. This isn’t simply how many people don’t complete the process, although that’s the number we seem to care about. Truth is, by itself it doesn’t tell you much about what might be wrong. A checkout process can be a number of steps, and each step is its own little mini-conversion. You really want to know where in the shopping cart folks get frustrated enough to say goodbye, because that’s the point where the cart’s conversion process is failing.

How Much Money’s Left on the Table?

We get lots of questions about how to solve the problem of shopping cart abandonment. But sometimes you don’t necessarily need an answer to this question – you need a better question.

Much better results, relatively speaking, can be yours through providing relevant and persuasive content throughout your Website: content that is based on an understanding of your visitors’ intentions and needs, content that instills confidence, content that reassures, content that makes an emotional connection. Visitors who have been thoroughly persuaded demonstrate uncanny motivation to navigate even the worst checkout process!

But, let’s face it. Money on the table is money that isn’t in your pocket. So, if you’ve got a tidy little cybershop and want to improve your shopping cart abandonment rates, then I’ve got any number of ideas for you. Next time!

click here for a printable version of this entire article

Feedback?

P.S. If you enjoyed this issue, why not share it with your colleagues and friends?
They'll appreciate it. Forward This Issue To A Friend!