Questions? (877) 643-7244
FutureNow Post
Thursday, Apr. 30, 2009 at 8:21 am

On a Scale From 1 to 5 Surveys Stink. Here’s Why!

By Jeff Sexton
April 30th, 2009

You know the kind of surveys I’m talking about, the ones that ask you to rate something on a scale of 1-5, they are called Likert surveys.  I doubt if anyone actually likes them, but I truly loath them.  Here’s why:

  • The rating system is too clunky. Most people get stuck between 3 and 4, usually with 4 sounding too good and 3 too wishy-washy, meaning that the results are often more indicative of a temporary mood than an honest difference.
  • Written answers are almost always more informative than raw numbers and everyone knows it, but they’re rarely asked for.  The words people chose, the way they phrase things, what they actually comment on, what details are mentioned, all add up to a much richer insight into the psychology behind the responses.  They provide context.  But Likert scales are rarely asked in conjunction with written responses and the overwhelming preference is for Likert scales over full responses.
  • Numbers are preferred over written answers because they’re easy – and easily averaged.  The reason organizations like Likert surveys is that the results are easily totaled and averaged.  You can express the results with mathematical certainty.  That’s harder to do with written responses.  So most organizations somehow decide that it’s better to be precisely wrong than approximately right.
  • The mind provides misleading answers to questions of the heart. Ever noticed how most respectable psychological research “tricks” the participants.  Participants are always told the experimenter is studying or looking for one thing, when it’s really something entirely different.  This indirection is considered necessary so that the participants self-conscious desires and biases don’t taint the results.  Likert-scaled surveys almost never use this technique.  Instead they ask direct questions about participants feelings, actions, and future actions.  And as Coke’s misstep with New Coke proves, the results of these surveys simply can’t be trusted.
  • No one bothers to write questions (and answers) in a psychologically astute manner. It usually helps to write questions and the attendant answers so that an honest response will not seem self-incriminating to the participant.  Ask a mom if she feeds her kids a lot of fast food, and you’ll probably get a false answer.  What kind of mom would answer yes?  Ask her if she frequently finds herself strapped for time and looking for food preparation and mealtime shortcuts and then follow that up with a question about the mom’s most used go-to solutions to food prep shortcuts, and you’ll get an entirely different outlook.*  Yet almost no one takes the time to do this with Likert-scaled surveys.  And so they get bullshit answers.  Go figure.
  • The results are almost always abused.  Surveys are as easily used to bolster a prejudice or further an agenda as they are to actually shed light on a subject.  Of course, any study can fall prey to this manipulation – if you torture the data long enough, you can get it to confess to anything – but the doubly abstracted nature of Likert survey results are far more easily abused than a compilation of written survey answers.  Want an example of this and most of the previous concerns?

Check out this little work of horror from Marketing Sherpa.  Let’s start with their interpretation of the survey and work backwards from there.  So here’s what they think their survey indicated:

“Two-thirds of marketers who work for organizations that have not used any form of social media marketing or PR consider themselves “very knowledgeable” or “somewhat knowledgeable” about this emerging strategy.  Their overconfidence in unproven ability can doom social media initiatives to failure.”

And what do they base this interpretation on?  A worse-than-normal Likert-scaled survey with only 4 badly worded answers.  Marketing Sherpa didn’t provide the exact question in the post, but it was centered on the respondents’ knowledge of social media marketing for organizations.  At any rate, here are the possible answers:

  1. Not knowledgeable at all
  2. Not very knowledgeable
  3. Somewhat knowledgeable
  4. Very knowledgeable

So think about it: you’re a marketer, maybe even specializing in interactive/internet marketing.  You’ve played around enough with social media to be comfortable with its dynamics and to know that most so-called social media experts aren’t, mostly because it’s an emerging field and few can claim legitimately successful social media marketing campaigns for non-entertainment or cutting-edge/sexy companies.  Then again, you know you’re no expert either.  So what do you select?

Not surprisingly 58% of the respondents selected “Somewhat knowledgeable.”  The survey basically forces you into that response unless you want to admit that you’re all but clueless about a rather important and emerging element of online marketing.  Even still, 28% of participants selected “Not very knowledgeable.” My guess is that if Marketing Sherpa had worded the choices more intelligently, avoiding the perception of self-incriminating answers, they would have had even more people falling between “not knowledgeable at all” and “somewhat knowledgeable.”

At any rate, the numbers show that 86% of respondents basically indicated that they are not totally clueless, but they aint all that, either.  Not exactly shocking answers given the question and possible answers.  And yet, this is the basis for Marketing Sherpa’s conclusion that the respondents were dangerously “overconfident.”  Give me a freakin’ break!

The real lessons of this?

Stay away from Likert-scales.  And especially avoid them when you’re trying to gauge people’s perceptions, feelings, ambivalencies, etc.  Do the real intellectual work of crafting intelligent and nuanced essay questions.  Invite open ended responses.  Comb through the answers with eye towards being approximately right rather than precisely wrong.

* Special thanks to the talented Holly Buchanon for sharing the McDonald’s survey example with me.

If you enjoyed this post please consider Tweeting it please.

Add Your Comments

Comments (52)

  1. Same goes for many multiple choice answers. Invariably my answer is never listed, so I’m forced to pick one, or skip the question. (Example from one “lifestyle” survey: City in which you’d most like to live. “My” much-loved “wouldn’t live anywhere else” city, Albuquerque, was not listed. Would have been simple enough to have “the city in which I’m living now” and “other” as choices.)

    Having worked in market research, I know you can tumble/rumble the numbers to get any result. “What do you want the result to be?” was the question to one particularly evil, contrary CEO. Ta-DA!

  2. I agree. I am firm believer that for valid answers we need to get better at monitoring behaviour rather than simply asking hypothetical questions.

    I wrote about this in my paper “Trust Measures and Indicators for Customers” (available at http://trustenablement.com/local/Trust_Measures_and_Indicators_for_Customers.pdf). See Trust Indicators on page 6.

  3. Great and timely post, Jeff. I actually took a survey last night! I would add that if you feel you have to do Likert scale questions, PLEASE keep them to a minimum, too. The survey I took had a strong incentive, but geez it took me about 25 minutes! Also, if you want to allow text responses, and are worried about the overhead of reading and interpreting them all, just limit the number of characters, which will force brevity.

  4. Agreed with your assessment of the Marketing Sherpa survey, there is certainly the introduction of bias due to the phrasing of the question. Many surveys, particularly online surveys, are poorly written or executed and the criticisms that you make in this column are due to these mistakes, not due to the weakness of the research method in general.

    The idea throwing out surveys is absolute blasphemy. Organizations everywhere are using properly modeled and executed survey research to develop marketing insight.

    Written answers are highly subject to bias on the part of the researcher (you emphasize positive responses), particularly if you don’t code responses into buckets. How would you best interpret a large quantity of written responses? Code them into buckets using a Likert scale (Very Negative, Somewhat Negative, etc.). The idea of numbers being able to be easily studied is exactly the point. How can you draw a conclusion without answering the question: “Is X different than Y?”

  5. I took one of these the other day that was quite different…. rather than provide an empty 1-5 scale and have the me pick the number – it was pre-populated with 3′s, or average satisfaction across the board. I could in one click submit and it would be average top to bottom.

    This made me really think about moving the selections up or down, or just leaving them at average 3′s.

    I moved one or two up or down, and left the rest alone. I think this would bait really meaningful results from the survey. There would be great data/trends from this approach, not just the actual results, but what questions resonated the most by having the most adjusted responses.

    A subtle but meaningful change of tactic, I my opinion.

  6. I don’t trust any polls or surveys these days. How many media outlets truly practice responsible journalism anymore? As with most polls and surveys, everyone has their spin, special interest or agenda they are trying to push.

    Gus
    Franchise Solutions

  7. Tom,

    I’m not advocating that marketers get rid of surveys, but that they favor written responses over numerical scales. Even properly modeled, scaled surveys simply don’t provide the level of insight as written responses. Are written responses subject to bias? Sure, but not nearly so much as numbers.

    As for bucketing written survey answers, I would much rather start from a written response that gives me relatively more insight into motivation, emotions, thought processes, assumptions, etc on the part of the survey participant AND THEN bucket the responses, than to have the participants bucket the answers and have their bucketing criteria hidden from me.

    As a copywriter, I have always found written responses to client surveys, or customer service transcripts and such to be incredibly helpful. I have almost always found client’s scaled surveys and the interpretations attached to them to be dangerously opaque, misleading, and ridden with assumptions.

    Is it theoretically possible to design a good Likert-scaled survey? Sure. But IMHO, the level of customer insight required to design one is so high that it would obviate the need for the survey in the first place.

    - Jeff

  8. Dan,

    The alternate model you mention is fascinating. I think seeing what answers people actually care enough to answer would be the most beneficial part of the survey at that point!

    Thanks for sharing.

    - Jeff

  9. Absolutely fantastic post.

    Taking just about any online survey these days is frightening as questions lead and answers are almost never broad enough to allow for true expression.

    I do feel that it’s important to include scales that can lead to overall metrics (trying to evaluate every long form response into a category yourself is not good) but that doesn’t mean whipping up a question without thinking it through.

    But when it’s all said and and done I’d rather ask 3 open ended questions and 1 scaled one than 50 scaled.

  10. I don’t completely agree that surveys stink. You mentioned some valid points including the fight between mind and heart but there is no proof to show that mind always wins. One cannot outright reject the survey base on rationals.

    Surveys have been a big source of almost all the mass researches.

    There are ways to overcome the psychological effects of answering likert scale questions -

    1. Adding a comment box at the beginning of each survey. This can reduce the number of responses but it will ensure that the likert scale answers are in line to the message in the comment box.

    2. Phone or personal conversation before the survey is taken.

    3. Asking the same questions in a different way to ensure the responses are in line.

    4. Asking questions that can cause interest in the process. Most surveys start by asking personal information. This puts the mind of the survey taker in “not again” mode. Questions can position the mind in a way that the entire survey process is enjoyable / controversial.

    5. Using statistical analysis to process the survey results. Averages / percentages can skew the data but having statistical control limits reduces the probability of error.

  11. Before you do any kind of survey you need to plan what you’re going to do with the results.

    Microsoft do a great job on their huge and growing support site. Each page has a mini-survey embedded into the end with just 2 questions:

    1) Did this solve your problem ? Y/N/don’t know

    2) Was this relevant ? Y/N

    Then a comment box: “What can we do to improve this info…”

    The results from this can be used to improve the support site.

    Starting with the high-traffic but low-satisfaction pages.

    And reading the comments – at the very least you start to know the vocabulary that ordinary people use to describe their problems and their emotions.

    Oddly enough I got a Microsoft survey yesterday and it was a horror story – pages and pages of questions – no clue about how many pages to come – and weird, irrelevant questions like

    “Has my overall view of the quality of microsoft’s product range increased after visiting the site – on a scale of 1-10″

    So they can be the best and the worst on the same day !

  12. The better surveys use various types of questioning (multiple choice, open end, Y/N, text open questions) as well as different scales in order to get the information needed and meet the goals of the survey

  13. Good post. Want another reason why 1to5 scales stink? They are one of the main reasons why Online Ratings and reviews lie.

    Here’s a fuller explanation:
    The Lies behind Online Ratings and Reviews

  14. On-line surveys in the e-commerce space would be better served by using the “Net Promoter Score” methodology, which asks one question, and rates on a scale of 1 to 10. No one has time or the inclination to answer pages and pages of questions. My company has been doing medical surveys for the past 12 years now, and no one wants to be burdened by a bunch of survey mumbo-jumbo when they are simply trying to make a purchase online.

  15. This article makes a lot of sense but there are some cases where having text boxes in surveys adds no value whatsoever. We are a content development company and we offer a free download “Journalism schools in India” on our website and we have a pre-download form. The form collects contact information and then we have a text box that asks journalism students to tell us why they are downloading the report. Most of them type things like “NA” or “Blah Blah Blah” instead of taking the time to explain their reasons. We are now planning to change the question type into a multiple choice question that lists a few reasons so the wannabe journalists can pick one. We could always solve the problem of ‘my choice is not listed’ by having an “other” option with a text box.

  16. Charlie,

    The saving grace for reviews, though, is that the numerical score is almost always accompanied by a write up. And, yes, I invariably learn more from the write up than the score. In fact, I actively discount the score unless the write-up strikes me as intelligent, sober, balanced, etc.

    On the whole, though, I’d have to agree that numerically scored reviews absent written information would be far less useful and, in many cases, misleading.

    - Jeff

  17. Nishi,

    I think you are potentially embarking on a very dangerous line of reasoning. The “NA” and “blah blah blah” responses indicate that the respondents are totally unmotivated to share their real thoughts with you. Making the format easier won’t change that! It is a FAR, FAR, FAR better thing to get a response that allows you to KNOW that the respondent didn’t care to give an honest response, than to get a response that you THINK is valid/sincere, but is nothing more than the product of an apathetic check in the box. You are far more likely to be misled by the check in the box. In fact, this is exactly what Dan Auns was getting at when he talked about the survey that was pre-filled out with 3′s. At least in that instance you could conceivably discount any 3 answer (and any response that was only 3s) and only look at what questions were important enough to evoke a changed score.

    My advice to you? Figure out a way to better motivate your survey participants to give thoughtful written answers rather than switching to a numerically scaled response.

    - Jeff

  18. Jeff, thanks for the advice. I do agree with you. But what do you think of the “Other” option with a text box? You are right- The problem is that people want to get to the download as quickly as possible and they see the question as a hurdle.

  19. Obviously the most important part of any survey is getting the questions right. That said, we’ve tried surveys to get feedback about the sites we design but never really learned anything useful. Recently, we did a user test with Feedbackarmy which allows you to create six questions and get written (i.e. open ended) responses from the users. Much more interesting (and actionable) results.

  20. I agree that you can learn more from reading a review rather than just looking at a star rating. After all a written answer can bring up all sorts of information that a rating or question and answer survey would not reveal.

    Although a rating of one to five may seem clunky, it is preferable to the alternative: percentage-based ratings.

    I run a review site for backpackers hostels and youth hostels and a few years ago I considered changing to a percentage-based rating system. My reasoning was that it would eliminate the delimma of choosing between a 3-star and 4-star rating when you really wanted to award a 3½-star rating. However the more I looked into it I noticed how a percentage-based rating system results in ratings skewed much higher than what you would get with a star-based rating system. I explain why in the following blog post: http://www.bug.co.uk/blog/2009/02/19/why-star-ratings-are-more-accurate-than-percentage-ratings/

  21. The central problem with surveys today is I think there are too many. It seems like every site I do business with asks me to do a survey when I start to leave. Now when I do a survey I often do it in a hurry. There is a benefit to them, but they are losing their impact.

  22. [...] # May 5th update:  article on why “On a Scale from 1 to 5 Surveys Stink.” [...]

  23. [...] On a Scale From 1 to 5 Surveys Stink. Here’s Why! [...]

  24. There is absolutely nothing misleading, incorrect or otherwise ‘wrong’ with using a Likert approach, if you both create it and evaluate it professionally. Psychometrics, the science of creating psychological tests, often use Likert scales, but in the context of questions which have been build out with an understanding of their convergent, internal and external validity.
    The use of any type of question style can be abused – but blaming the format of the scale is well off the mark. The point of asking a lot of people something is to try and determine the average way people feel about something – asking an open ended question allows you to see how one person feels about something – the exact opposite of what a survey is supposed to do, and hence, a perfectly pointless rebuttal for the misuse of a particular type of scale.
    And… you know, you just can’t perform a Chi-square calculation on an essay question.

  25. Likert scales can and are used very effectively. What you seem to be arguing is the poor implementation of Likert scales.

    In nearly every application you can do it poorly or do it well. I could point to hundreds of thousands of blogs that are abysmal, but I wouldn’t say that blogs aren’t worth doing.

    Open ended questions are open to the same type of question bias and can be written in ways that will generate responses of a certain type.

    Likert scales can be used to consistently track behavior and attitudes over time. I’d argue that open ended questions make this far more difficult.

    Are you doing a keyword analysis of the responses? How do you interpret the open ended results? Do you interpret them the same from time period to time period? There’s a lot more subjectivity in open ended questions is there not?

    You may take a person’s comment out of context, or that person may not have accurately communicated what they meant.

    Using both Likert scales and open-ended questions is a better strategy. Throwing the baby out with the bath water just doesn’t make much sense to me.

  26. AJ Kohn & Naoise,

    Just to establish some context, grokdotcom is an online persuasion and marketing blog, speaking to owners of Internet businesses and other online/internet marketers and professionals. So my comments quite admittedly do NOT apply to the use of Likert-scaled surveys within various academic fields and specialties such as psychology, sociology, etc.

    So, for the audience in question, to say that I’m only talking about the abuse of Likert scales may be technically true, but is misleading as most people think that abusive practices represent a minority, where as probably more than 95% of Likert scaled surveys used within marketing, PR, customer research, etc represent misuse/abuse of the tool.

    I’ll put it this way, I’ve seen hundreds of various scaled surveys and I’ve never seen one that provided valuable insight. Almost every open-ended survey or even customer service transcripts I’ve come across have yielded gold. Even mediocre open-ended surveys typically have nuggets.

    That either type of survey can be abused (or used effectively) misses the point. In actual use Likert-scaled surveys (without any open ended or written add-ons) are:

    * vastly more likely to be poorly constructed
    * far less tolerant of sloppy construction
    * far more opaque in the answers they yield
    * far more likely to have their numerical, and therefore seemingly “objective,” results represented as “fact”

    Given the reality on the street when it comes to these surveys, I stand by my advice to favor open-ended questionnaires. For those with expertise in creating Likert-scaled surveys with “an understanding of their [questions] convergent, internal and external validity,” than I’d say, have at it. But that doesn’t even remotely describe 99.9% of marketers, customer research teams, etc.

    - Jeff

  27. [...] http://www.grokdotcom.com/2009/04/30/on-a-scale-from-1-to-5-surveys-stink-heres-why/ Note that this is about Likert-scale surveys, not all surveys. Indeed, the author’s main beef is that they’re not qualitative – just the opposite of 4Q visitor surveys. [...]

  28. I think the issue here is with the question. It asked for self-ranking. That’s just silly. Eg. A question such as “Are you a handsome man?”; most will go for the middle ranks as they don’t want to seem egotistical nor insecure. Likert method works for simple and specific question like: How do you rate our front desk service?

  29. Very well argued Jeff. But you could go further – the wider issue is that just like graphic design tools encourage people with no design skills whatever to think they can “design” something themselves, easy survey tools encourage people with no knowledge of statistics or research techniques to think that they can do surveys. Whch if course they can, but so badly there is usually worse than no point.

    We could go on forever, but the main thing I’d like to add is the issue of sample and population selection. Almost all surveys these days seem to be carried out so unscientifically that it probably doesn’t matter what questions you ask, the results will be invalid. I saw a survey last week that claimed that over 50% of businesses were actively using Twitter for their business! For crying out loud, 50% of all businesses aren’t even online yet. They’d surveyed a mailing list subscribed to mostly by digital marketing agencies, and assumed the results would be good for the whole business world.

    What I like about your approach is that it generates qualitative responses, which can be examined individually. I’m not sure the question format in your examples is the real problem, I’d say that was lack of expertise in surveying. But unless you are a well trained survey professional, qualitative is the way to go. It’s just a safer weapon in inexperienced hands.

  30. [...] New on the GrokDotCom “Top Marketing Optimization Posts” newsletter comes an article by Jeff Sexton titled “On a Scale of 1 to 5 Surveys Stink. Here’s Why!” [...]

  31. A factual correction is in order “The reason organizations like Likert surveys is that the results are easily totaled and averaged.” No experienced researcher relies on averages from these Qs. We typically report and compare “Top 2″ (or if relevant, “Bottom 2″) boxes because obviousy averages are often too muddy.

    Still–some of your comments are quite correct. But in reality, an experienced market researcher will often recommend a combination of quant and qual methods (depending on goals and budget); the default in quant research is not a questionnaire built largely around Likert scales.

  32. Here is a great website: Research Wall of Shame. Some very funny examples of how not to do research.
    The title suggests “everyone thinks they can do research, but not everyone should”. You can even post your own crazy examples there and share with everyone.

  33. Likert scale was meant to scale down the level of agreement to a statement. For me, it didn’t serve the real intention of collecting respondents opinion as the answers will always too general and narrow. Like you’ve said, most people will get stuck between 3 – 4 as we tend to be not to optimistic in many things. So the whole research concept will become pointless as we already know what the results would be.

  34. Yes I agree some surveys can be abused, but overall most sites that conduct surveys are legit and do infact pay for you opinion, so with that being said I’d say that most survey sites have measures in place to ensure survey accuracy data. I am a member of many of these surveys sites you speak roughly about and can say data in most cases is not compromised. You can check out more at my legit survey site reviews blog. I list the top survey sites that pay. Believe you me they take every measure to ensure that surveys are fun easy to fill out.

  35. I read a couple of days ago on a report from Youtube that most people rate the videos either 1 staror 5 stars, so they are considring to remove the starred rating system.
    I’m also thinking on implementing a rating system on shpallje but maybe it will be only thumbs up/down, instead of stars.

  36. Maybe instead of removing the rating system, Youtube should consider displaying a percentage value for those interested (you click on the rating to see the percentages). For eg. 51 percent rated it 3, 23 percent rated it 2 etc. Should not be really hard to do.

  37. Great Post,I agree.

  38. Great article
    i think there are a lot of problems with surveys today.
    Once i am gonna leave the site i do business,it sks me to do a survey.

  39. I agree totally with you, I cruise about 3 times a year and we get a survey on board and another in the post on return, they never give you the in between option, and very rarely can you make comments.
    I also get stuck between 3 and 4, it is so annoying.
    I would rather have a comment box, yoiu can then say what you want.

  40. [...] On a Scale From 1 to 5 Surveys Stink. Here’s Why! [...]

  41. good post, keep it going on

  42. Could your online surveys be better and yield more data? Check out these professional strategies http://snurl.com/smartsurveys

  43. Jeff, your articles are brilliant and informative, thanks a ton.

  44. I always knew surveys were total BS.

  45. Is there anything more boring than a web survey? I think not!

  46. Is there anything more boring than a web survey? I think not!

  47. Nice Article, I never thought before, It’s very useful. Thanks

  48. Written answers are helpful but they can also be useless and can be misinterpreted too. People say one thing and do another. People say one thing but mean something else. I’ve seen this recentlty in a usability listening lab for an online retailer. The subject said that she felt distracted by the product images that showed the faces of the models vs. the ones with cut off heads.

    If we take what she says at face value then we would believe that our current practice of showing products on models with their heads cut off was still a good idea.

    But she was really distracted by the variety of poses and facial expressions. She just didn’t know how to articulate this.

    I’m currently reviewing open ended survey questions and a lot of what people write is useless.

    Likert scales aren’t inherently bad. It’s about taking the care to write good surveys that adhere to research best practices and having skilled people to the analysis.

  49. Nice theory! I always stuck on Likert surveys answers..they all to much alike. LOL

  50. Is there anything more boring than a web survey? I think not!

  51. I agree that written responses would be better but that often isn’t an option. Most surveys would not be possible if they couldn’t gather answers quickly and economically. I do see your point and agree that you must see the questions/answers before you can believe the results.

  52. 1-5 gets you a small point of view, but when you add comments it gives you more of a human perspective. but if you’re doing mass scale surveys it becomes difficult to self analyze.

Add Your Comments

 

Print this Article
Share

Jeff is a Persuasion Architect, Web copywriter, blogger, and instructor of FutureNow's Persuasive Online Copywriting workshop. Follow Jeff Sexton on twitter

More articles from Jeff Sexton

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: