The survey involved nearly a thousand participants ages 16 to 24, and the good news for us as online marketers was that 75% of respondents said that they “couldn’t live without the Internet.” Wow, that’s a bold statement indicating that the Web will continue to be a central part of that generation’s lives, and will be essential to how they research, form opinions, make decisions, and buy.

Maybe it’s not “bad news,” but the challenge this presents is: How will our generation of digital marketers and online businesspeople keep up with the attitudes and expectations of a new segment that doesn’t remember the absence of the Web?

"Kurt & Smitty" | Image from AMCTV.com

I can’t help but think of my latest TV series obsession, Mad Men.  The fictitious ad agency, Sterling Cooper, hired two young upstarts, Smitty and Kurt, to help them figure out what the youth of the early 1960s was thinking.  They informed their hopelessly old employers that the youth “don’t want to be told what to do or how to act. [They] just want to be.“  Sound familiar?  It sounds corny, but Sterling Cooper did the right thing by bringing in outside perspective to help them better empathize with their target audiences.

So, what are we going to do? Should all marketing teams have their own “Smitty and Kurt” to help them grok how Digital Natives want to interact with businesses?  Should we spend more time spying on our children and grandchildren as they network their always-available lives at lightning speed?  Let us know in the comments what specific tactics you’re using to keep up with what “the kids are into these days.”

• The rating system is too clunky. Most people get stuck between 3 and 4, usually with 4 sounding too good and 3 too wishy-washy, meaning that the results are often more indicative of a temporary mood than an honest difference.

• Written answers are almost always more informative than raw numbers and everyone knows it, but they’re rarely asked for.  The words people chose, the way they phrase things, what they actually comment on, what details are mentioned, all add up to a much richer insight into the psychology behind the responses.  They provide context.  But Likert scales are rarely asked in conjunction with written responses and the overwhelming preference is for Likert scales over full responses.

• Numbers are preferred over written answers because they’re easy – and easily averaged.  The reason organizations like Likert surveys is that the results are easily totaled and averaged.  You can express the results with mathematical certainty.  That’s harder to do with written responses.  So most organizations somehow decide that it’s better to be precisely wrong than approximately right.

• The mind provides misleading answers to questions of the heart. Ever noticed how most respectable psychological research “tricks” the participants.  Participants are always told the experimenter is studying or looking for one thing, when it’s really something entirely different.  This indirection is considered necessary so that the participants self-conscious desires and biases don’t taint the results.  Likert-scaled surveys almost never use this technique.  Instead they ask direct questions about participants feelings, actions, and future actions.  And as Coke’s misstep with New Coke proves, the results of these surveys simply can’t be trusted.

• No one bothers to write questions (and answers) in a psychologically astute manner. It usually helps to write questions and the attendant answers so that an honest response will not seem self-incriminating to the participant.  Ask a mom if she feeds her kids a lot of fast food, and you’ll probably get a false answer.  What kind of mom would answer yes?  Ask her if she frequently finds herself strapped for time and looking for food preparation and mealtime shortcuts and then follow that up with a question about the mom’s most used go-to solutions to food prep shortcuts, and you’ll get an entirely different outlook.*  Yet almost no one takes the time to do this with Likert-scaled surveys.  And so they get bullshit answers.  Go figure.

• The results are almost always abused.  Surveys are as easily used to bolster a prejudice or further an agenda as they are to actually shed light on a subject.  Of course, any study can fall prey to this manipulation – if you torture the data long enough, you can get it to confess to anything – but the doubly abstracted nature of Likert survey results are far more easily abused than a compilation of written survey answers.  Want an example of this and most of the previous concerns?

Check out this little work of horror from Marketing Sherpa.  Let’s start with their interpretation of the survey and work backwards from there.  So here’s what they think their survey indicated:

“Two-thirds of marketers who work for organizations that have not used any form of social media marketing or PR consider themselves “very knowledgeable” or “somewhat knowledgeable” about this emerging strategy.  Their overconfidence in unproven ability can doom social media initiatives to failure.”

And what do they base this interpretation on?  A worse-than-normal Likert-scaled survey with only 4 badly worded answers.  Marketing Sherpa didn’t provide the exact question in the post, but it was centered on the respondents’ knowledge of social media marketing for organizations.  At any rate, here are the possible answers:

1. Not knowledgeable at all
2. Not very knowledgeable
3. Somewhat knowledgeable
4. Very knowledgeable

So think about it: you’re a marketer, maybe even specializing in interactive/internet marketing.  You’ve played around enough with social media to be comfortable with its dynamics and to know that most so-called social media experts aren’t, mostly because it’s an emerging field and few can claim legitimately successful social media marketing campaigns for non-entertainment or cutting-edge/sexy companies.  Then again, you know you’re no expert either.  So what do you select?

Not surprisingly 58% of the respondents selected “Somewhat knowledgeable.”  The survey basically forces you into that response unless you want to admit that you’re all but clueless about a rather important and emerging element of online marketing.  Even still, 28% of participants selected “Not very knowledgeable.” My guess is that if Marketing Sherpa had worded the choices more intelligently, avoiding the perception of self-incriminating answers, they would have had even more people falling between “not knowledgeable at all” and “somewhat knowledgeable.”

At any rate, the numbers show that 86% of respondents basically indicated that they are not totally clueless, but they aint all that, either.  Not exactly shocking answers given the question and possible answers.  And yet, this is the basis for Marketing Sherpa’s conclusion that the respondents were dangerously “overconfident.”  Give me a freakin’ break!

The real lessons of this?

Stay away from Likert-scales.  And especially avoid them when you’re trying to gauge people’s perceptions, feelings, ambivalencies, etc.  Do the real intellectual work of crafting intelligent and nuanced essay questions.  Invite open ended responses.  Comb through the answers with eye towards being approximately right rather than precisely wrong.

* Special thanks to the talented Holly Buchanon for sharing the McDonald’s survey example with me.

If you enjoyed this post please consider Tweeting it please.

According to the Omniture survey:

• Marketers picked cost per click and click through rate as among their top metrics to optimize search campaigns instead of deeper metrics such as return on ad spend, cost per customer (or sale) or profit per order
• 43 percent of e-commerce respondents do not know how to accurately measure profit per customer (or order)
• 67 percent of respondents indicated not having enough time to effectively manage campaigns as their top issue in search marketing, while only 35 percent use an automated bidding solution

Are you surprised? We aren’t; check out the 7 Deadly Sins of Web Analytics.

Every day leading companies strut their James-Bond-cool stuff, showing off their sophisticated tools and fancy talk about ROI.

It’s hard to take them seriously knowing that underneath that James Bond tuxedo their diapers are soiled.

In 1998 we were frustrated because companies didn’t understand their conversion rates. In 2009 we remain frustrated.

Much less than 1% of the thousands of companies we’ve spoken with are the exception.

Do you have any insight as to why marketers remain enamored of the shiny new object but reject focusing on the fundamentals?

___________________________________

P.S. Josh James, the CEO of Omniture, in January of 2005 said “Web analytics can pay for itself with a single business improvement — so the real question is how quickly can companies make data-driven decisions? This willingness to change will ultimately dictate time to ROI.”

The “Voice of the Customer” (VOC), can be obtained in many ways: surveys, reviews, customer requests, interviews,  focus groups, field reports, etc. In order to find those golden nuggets that can lead to improvement you need to start with the segment of customers that like you the least.

In the first case, those that like you the least are more likely to be biased in their observations; that is, there are of course people who actively dislike your company or products or services, though the stronger they feel so the less likely they will respond to a VOC appeal anyway. Rather, those responding to a VOC appeal would be those who “unlike” you: they’ll be quasi-neutral, perhaps ticked-off at some silly thing you’ve done, a mild dislike at worst. Thus, they will also tend to improve their feelings toward you as soon as you engage in an attempt to listen to them, and you’ve given them a chance to vent. Especially about stuff that you’re blind to. So VOC surveys that are meant to reinforce what you think or feel without some a mechanism for real painful insights into your systems’ flaws tend to not yield much benefit.

Voice of customer programs have picked up in adoption in the last couple of years, especially in the last twelve months with free options like 4Q, Kampyle, and GetSatisfaction. Of course you can also use tools like surveymonkey, zoomerang and others to launch surveys as well. Paid options include Foresee, iPerceptions and Opinion Labs.

Since there are so many tools already available, I am going to ignore issues involving “setup” of a VOC solution, and start instead with an exploration of various invitation-to-participate options.

Option 1: Intercept on Arrival – This approach is meant to engage visitors before they interact with your website and have any set expectations. The way this gets launched is typically by some random sample (although I am still seeing too many sites using this on 100% of their visitors – and that is a bad practice) and it presents an invitation to provide feedback after they finish their experience on the website. This can be felt as intrusive to a segment of your audience, especially repeat visitors if the survey invitation keeps popping up (because the cookies that are set were not found again). 4Q works this way. 4Q’s research has shown an increase in conversion and brand impression by using VOC with this method. It would seem to indicate that a website that is listening to its audience seems to instill greater trust in the brand, or at least  it gains more from the listening than it generates in additional irritation from the “in-your-face” interception.

Option 2: Intercept on Action/Behavior – This approach offers survey recipients to engage with a survey based on their previous actions on the website. An example could be launching a survey when someone abandons a shopping cart. These surveys are insightful only towards that limited task and not your audience as a whole, but may provide you with tactical and actionable recommendation on resolving particular task issues encountered. This is can also feel intrusive, and if someone is already dissatisfied with a brand interaction and you pop-up this survey it may feel like rubbing salt into the wound. When it’s done, it has to be implemented with the lightest of touches.

Option 3: Passive – While a passive invitation is non-intrusive to the customer experience, it tends to favor toward those who favor actively providing feedback. Often you will find this as a embedded link, a wdiget in the corner of a page, etc. People who have had a negative experience tend to be the ones who seek out these feedback mechanisms, and are usually used to deal with more tactical issues that occur on the page level. Those who have had positive experiences tend to not leave as much feedback with this method, which causes us to often misunderstand the size and scope of the issue. Response rates tend to be lowest in this format. There’s also a sample bias towards buying modality, insofar as Humanistic personas will make up a larger proportion of this response group as compared to your general audience.

Each of the above options is viable; they are all worthwhile tools for the right job. The important piece to remember is that web analytics is meant to show us the what has happened and VOC is intended to help illuminate why. This is the reason it is important to tie analytics and VOC tools together. Voice of customer is driven by the need to have actual customer feedback woven into your future customer interactions. You can also collect feedback based on what people click and interact with on your website as well (think buttons in a flash demo, filling in a calculator, etc).

The reason we want to collect this information is because we want our customers to have greater satisfaction, an improved experience, and a visit where they achieve what they came to accomplish. The insights provided by VOC should help us in our continuous improvement efforts by helping us align our goals with the customers’ goals and identifying possible friction points.

Voice of the Customer programs are meant to capture the open-ended dialog, because that is where we often see the deeper insights. Like every analytics approach, you gain the most when you can segment by areas that you have already identified as potential weak points through the use of other analytics metrics or usability studies.

In fact, 4Q has just released some segmentation features to their free survey tool and plan on adding additional ones (full disclosure: I am an advisor to iPerceptions, the company behind 4Q). You can also do this by offering a segment-specific survey at a given point in their experience using one of the appraoches outlined earlier in this article observe what possible solutions to the problem may be uncovered.

Next post I’ll cover research design and what kind of questions are best to ask.

The challenge of web analytics is that while many of the tools are great at presenting the symptoms you still have to diagnose yourself.

Go ahead and ask any physician how sites like WebMD have impacted their practice and they will give you an ear full about people self diagnosing. While their view may be too restrictive it does often lead to self-medication and the problems lingering longer than they should.

Please help us get a better understanding of how you are using web analytics by taking this 6 questions, 3 minute survey. Last time we ran this survey, we found that 96.85% of people are running web analytics, and out of those, only 26.56% have a full time analyst and thankfully 53.12% are already testing.

Please feel free to pass along the survey to a friend or colleague. Thanks.