You know the kind of surveys I’m talking about, the ones that ask you to rate something on a scale of 1-5, they are called Likert surveys.Ā I doubt if anyone actually likes them, but I truly loath them.Ā Here’s why:
- The rating system is too clunky. Most people get stuck between 3 and 4, usually with 4 sounding too good and 3 too wishy-washy, meaning that the results are often more indicative of a temporary mood than an honest difference.
- Written answers are almost always more informative than raw numbers and everyone knows it, but they’re rarely asked for.Ā The words people chose, the way they phrase things, what they actually comment on, what details are mentioned, all add up to a much richer insight into the psychology behind the responses.Ā They provide context.Ā But Likert scales are rarely asked in conjunction with written responses and the overwhelming preference is for Likert scales over full responses.
- Numbers are preferred over written answers because they’re easy – and easily averaged.Ā The reason organizations like Likert surveys is that the results are easily totaled and averaged.Ā You can express the results with mathematical certainty.Ā That’s harder to do with written responses.Ā So most organizations somehow decide that it’s better to be precisely wrong than approximately right.
- The mind provides misleading answers to questions of the heart. Ever noticed how most respectable psychological research “tricks” the participants.Ā Participants are always told the experimenter is studying or looking for one thing, when it’s really something entirely different.Ā This indirection is considered necessary so that the participants self-conscious desires and biases don’t taint the results.Ā Likert-scaled surveys almost never use this technique.Ā Instead they ask direct questions about participants feelings, actions, and future actions.Ā And as Coke’s misstep with New Coke proves, the results of these surveys simply can’t be trusted.
- No one bothers to write questions (and answers) in a psychologically astute manner. It usually helps to write questions and the attendant answers so that an honest response will not seem self-incriminating to the participant.Ā Ask a mom if she feeds her kids a lot of fast food, and you’ll probably get a false answer.Ā What kind of mom would answer yes?Ā Ask her if she frequently finds herself strapped for time and looking for food preparation and mealtime shortcuts and then follow that up with a question about the mom’s most used go-to solutions to food prep shortcuts, and you’ll get an entirely different outlook.*Ā Yet almost no one takes the time to do this with Likert-scaled surveys.Ā And so they get bullshit answers.Ā Go figure.
- The results are almost always abused.Ā Surveys are as easily used to bolster a prejudice or further an agenda as they are to actually shed light on a subject.Ā Of course, any study can fall prey to this manipulation – if you torture the data long enough, you can get it to confess to anything – but the doubly abstracted nature of Likert survey results are far more easily abused than a compilation of written survey answers.Ā Want an example of this and most of the previous concerns?
Check out this little work of horror from Marketing Sherpa.Ā Let’s start with their interpretation of the survey and work backwards from there.Ā So here’s what they think their survey indicated:
“Two-thirds of marketers who work for organizations that have not used any form of social media marketing or PR consider themselves āvery knowledgeableā or āsomewhat knowledgeableā about this emerging strategy.Ā Their overconfidence in unproven ability can doom social media initiatives to failure.”
And what do they base this interpretation on?Ā A worse-than-normal Likert-scaled survey with only 4 badly worded answers.Ā Marketing Sherpa didn’t provide the exact question in the post, but it was centered on the respondents’ knowledge of social media marketing for organizations.Ā At any rate, here are the possible answers:
- Not knowledgeable at all
- Not very knowledgeable
- Somewhat knowledgeable
- Very knowledgeable
So think about it: you’re a marketer, maybe even specializing in interactive/internet marketing.Ā You’ve played around enough with social media to be comfortable with its dynamics and to know that most so-called social media experts aren’t, mostly because it’s an emerging field and few can claim legitimately successful social media marketing campaigns for non-entertainment or cutting-edge/sexy companies.Ā Then again, you know you’re no expert either.Ā So what do you select?
Not surprisingly 58% of the respondents selected “Somewhat knowledgeable.”Ā The survey basically forces you into that response unless you want to admit that you’re all but clueless about a rather important and emerging element of online marketing.Ā Even still, 28% of participants selected “Not very knowledgeable.” My guess is that if Marketing Sherpa had worded the choices more intelligently, avoiding the perception of self-incriminating answers, they would have had even more people falling between “not knowledgeable at all” and “somewhat knowledgeable.”
At any rate, the numbers show that 86% of respondents basically indicated that they are not totally clueless, but they aint all that, either.Ā Not exactly shocking answers given the question and possible answers.Ā And yet, this is the basis for Marketing Sherpa’s conclusion that the respondents were dangerously “overconfident.”Ā Give me a freakin’ break!
The real lessons of this?
Stay away from Likert-scales.Ā And especially avoid them when you’re trying to gauge people’s perceptions, feelings, ambivalencies, etc.Ā Do the real intellectual work of crafting intelligent and nuanced essay questions.Ā Invite open ended responses.Ā Comb through the answers with eye towards being approximately right rather than precisely wrong.
* Special thanks to the talented Holly Buchanon for sharing the McDonald’s survey example with me.
If you enjoyed this post please consider Tweeting it please.