Questions? (877) 643-7244
FutureNow Post
Friday, May. 29, 2009 at 9:14 am

Top 6 User Testing Mistakes and How to Avoid Them

By Jeff Sexton
May 29th, 2009

About a year ago, Bryan Eisenberg gave an interview talking about Maybe The Best $100 You’ve Ever Spent, essentially raving over the ridiculously cheep rates charged by UserTesting.com.  The always-astute Patrick Sullivan, Jr. of Edit Weapon picked up on this and decided to give UserTesting.com his own personal test and blog post/review.

Now, as a usability expert himself and a usability testing veteran, Edit Weapon’s initial reaction to UserTesting.com was:

Well this will either put me out of business, cause me to cut my rates by 90%, or make my life 900% easier!

The reasoning behind the first two reactions is obvious, but I bet me than a few viewers wondered how ultra-cheap (and effective) competition could possibly make Patrick’s life 900% easier?

Answer: because Patrick’s job isn’t primarily to provide user testing, but to help properly task the users and to expertly interpret the results of that testing.  Turns out that actually conducting the tests was just a pre-requisite to these far more important – and less easily commoditized – skills.

So offloading the pain-in-the-butt process of sourcing the testers and running the tests to UserTesting.com has made Patrick’s life a lot easier.  [Note that, in my opinion, that insight into what business Patrick is really in is worth a series of blog posts of its own, but that'll have to wait for another time...]

As Patrick put it in his blog post:

“…anyone can watch a user use a website, but *interpreting* usability tests and making recommendations is the secret sauce to being a kick ass information architect / interaction workflow designer, which of course, I am.”

Now, during that same video post, Patrick offhandedly mentioned that there were a few golden rules and guidelines to tasking users so that their test results would be optimally useful and easy to interpret, but that he’d have to cover these in a follow-up post.

Ever since hearing that I’ve been patiently waiting for Patrick to finally produce that promised follow-up post, until about a month ago when I broke down and offered to help with the post by turning it into a quasi-interview.  So here they are, the Top 6 User Testing Tips as disclosed to me by Edit Weapon:

Top 6 Don’ts for Usability Testing

(With a special thank you to Sue Fischer – Patrick’s IA/Usability/learnability mentor and a human factors consultant who taught Patrick how to task users for usability tests and how to interpret the results.)

1)    Never ask, “What do you think about this?”

First of all, most people will simply give you a polite, rather than bluntly honest answer.  Second of all, you’re not really interested in what they think of an interface/Web design/piece of software; you’re interested in how well and how easily they can USE it.  That’s why it’s called usability testing.

So you always want to put the question in the form of a goal/task.  Tell the user what they want to do with the interface/software; give them an assigned scenario.  This transforms the process into an objective exercise (rather than a subjective opinion) and allows you to watch how the testers go about using your tool.  You can then get a much better idea of how easy or intuitive your interface is, where the friction occurs, etc.

2)    Don’t feed the tester with your question.

As people learn new things they tend to be very literal – especially when it comes to tasking.  If you ask people to accomplish a task and you use the exact words or phrases that are actually ON the interface labels, you’ll wind up with a false impression of how usable your interface is.

For example, if you ask a tester to “Compose an e-mail” and the button for writing a new e-mail is actually labeled “compose e-mail,” the tester will simply match the phrases up rather than thinking organically in terms of what they’re trying to accomplish and then figuring out the interface. This is “leading the tester” by “feeding” him/her information with your questions.

So you want to ensure that you ask question using terms that are not directly on the interface labels.  Use synonyms.  Don’t make your tasks so easy that the tester simply has to match up terms.  Going a step further, if most users won’t think of a task in terms of multiple steps, but your interface requires multiple steps, don’t break your tasking down into steps to match the interface.  Write the question or task in the way that most users would think of it within a given scenario.

3)    Don’t let users be the designers.

When you get goal-oriented tasks, each user will have different levels at which they learn the interface and pick it up, and some users will do crazy things.  So some users will offer suggestions.  Don’t take those suggestions literally or at face value.  You’re looking for what users DO more than what they say.  This is similar to the rule against not asking users what they “think” of an interface.

4)    Don’t let the statistics fool you.

If you’ve done 20 tests in a row and, let’s say, 5 out of 20 were failures, but as you’ve been working on it and creating new iterations, the last 5 tests went extremely smoothly, you’ve got a good design.  You need to think of the results in terms of being 5 for 5 rather than 15 for 20.

This also applies to individual tasks within a test.  If users find some minor tasks are more difficult to accomplish than operating the really commonly-used features, don’t let those “usability problems” count anywhere near as much as your successes with the main functions of your interface.

Basically, not everything can be a big red button in the middle of the screen.  You have to balance things out and sometimes a few items are a bit more difficult to find and there’s really no perfect solution for a multiple use interface.

5)    Don’t get discouraged.

Your expectations are usually going to be high prior to the first test.  You’ll wonder how people aren’t seeing what they are supposed to see.  So user testing can be a humbling experience.

That’s why it’s usually best to test and tweak your interface in iterations.  You can’t design perfectly from the get-go because you are too much inside the bottle as the designer.  But as you alternate insight generated from testing with new and improved interface iterations, you’ll find the magic if you’re willing to hang in there.

6.    Don’t try to test too much at once.

You’ll get easier to analyze results if you limit your tasks to just 2-3.  And at UserTesting.com’s prices, it’s not a big deal if you end up running additional tests instead of adding more tasks to the same test.

But wait, there’s more… Patrick also walked me through how these principles played out when used to evaluate Jigsaw Health’s landing page for Magnesium Supplements.  Catch the walk-through on our next follow-up post.

Any tips, tricks or traps you want to share?

[Editor's note: the author of this post is now blogging at jeffsextonwrites.com]

Add Your Comments

Comments (65)

  1. Great tips. I am trying to use
    http://www.uxbooth.com/blog/userfly-usability-testing-made-easy/ for our website. Have you used it and what is your opinion about it.

  2. Just have to say that I 100% agree with Patrick. I was a little worried about these tools at first but then realized how much opportunity they represent for the UX world. It’s not only giving me an extra avenue for consulting, but has opened up testing for my small clients with limited budgets. In some cases, I’ve even run a 5-user test out of my own pocket just to get insights and make an impact on a client’s bottom-line. That’s not to say that a $29 remote user can replace a full-blown laboratory test, but it never hurts to have one more tool in your belt.

    Also, I love #3 in your list – it’s one of the reasons I try to put a layer of interpretation between test subjects and clients. THere’s nothing worse than when one 1 user says “That blue color sucks! I like red” and a client suddenly decides they want to do a complete redesign. Some people just don’t like blue – we’ve got to learn to live with it.

  3. #2 is key, but 3 was also good new advice for me. #1 will always be the first rule! thanks for the list.

  4. So Jeff what are the best things to put in a usertesting.com request?

  5. I have run tests with usertesting.com at least 4 or 5 times and I always found something new and useful to improve on. I really like them. This motivates me; maybe I will put in a request for another test?

  6. I was completely surprised by how eye opening these tests can be. Designers, marketers and the website owners are too familiar with a website to see them the way a new user does. Give them a try. You’ll be amazed!

  7. What I have seen is only 2% of the companies are trying to test the layouts or their landing pages for better results..

  8. My tip for usability testing:

    Give the subject a dummy task first to warm her up. Don’t say it’s a dummy task.

    My favourite (in the UK) is “A friend lives in Bristol. Go to the BBC website and find the weather forecast for Bristol today. Talk me through the process, please.”

    This helps to break the ice and get her used to describing what she is doing and why. It also helps you to calibrate her web skills.

  9. Usability testing is a serious investment of time and resources for any team. Having a clear understanding of what you want to get from it is critical to its success.

  10. Jeff, thanks for doing that interview with me.

    Correction:

    > If you’ve done 20 tests in a row and, let’s say, 5 out of 20 were failures…

    I actually meant, “let’s say only the last 5 tests out of 20 were successful and the first 15 all sucked, but you were making iterative changes all along…well the last 5 successful tests are saying you’ve got a usable design.”

    John (Site Doublers), I *LOVE* your “dummy task” idea! Will totally use that.

  11. Looks like a great service. Another one worth checking out is http://www.Loop11.com. This one lets you run your own online, unmoderated user testing.

  12. [...] Top 6 User Testing Mistakes and How to Avoid Them [...]

  13. 6 down and dirty e-marketing tips (Small Business E-commerce Link Digest – June 12, 2009)…

    Social media marketing and testing techniques top the trends in this week’s links. Check ‘em out.
    ……

  14. Jeff and Patrick, I’m with you on numbers 2-6, but I would take issue with the idea that you can’t ask the user’s opinion during a usability study (#1).

    Of course, the primary focus of a usability study must be on task completion, but eliciting users’ subjective responses to key content or functional elements is important to clarify their perceptions and give the project team a chance to hear what their users like, value and need. As long as these issues are explored at the _conclusion_ of a task, they don’t interfere with the ability to get solid observational data.

  15. Lyman, I agree with you, as long as the question is asked *after* the test. My point for #1 is that asking “what do you think about this?” is NOT a usability test.

    Btw, I always ask — at the completion of the test — “Please rate that on a scale of 1 to 5, with 5 being the best.”

    Good comment.

  16. [...] Top 6 User Testing Mistakes and How to Avoid Them [Grokdotcom] [...]

  17. Great advice!

  18. [...] About a year ago, Bryan Eisenberg gave an interview talking about Maybe The Best $100 You’ve Ever Spent. Read more.. [...]

  19. Great advice thank you

  20. I believe that the “kick Ass” user test will get FAR more out of a usability test then task completion alone. We test brand impact, anxiety, frustration and competitive sites as well, just for a start. Asking what people “think about a site” is usually just a starting point for deeper discussions and if you understand body language, their body will tell you what they really think, regardless of what their words say. Body language doesn’t lie. There is SO much more to user testing… I am glad that people are getting a test of the power of user testing but learning far more advanced techniques yields far more helpful results.

  21. [...] Do usability testing.  Get someone outside your industry (go ahead and specify minimum industry knowledge in your user request) and watch them move through your site while recording their questions, thoughts, etc. [...]

  22. Probably number #5 is what describe me really well..great read buddy!

  23. It is even true!
    Sometimes we are easily fooled by statistics.
    But I think the majority is misinterpretation of data.
    Thanks.

  24. I think that you can think about what your users want before you design something.. Even if you make it as you wish you should listen to their needs. feedback is always important in these situations.

  25. I love using the “Coffee”-test.

    Meaning sit down at your local Starbuck and offer a coffee in return for testing your website.

    It is cheap – and the user is not prepared (meaning they are giving you a more honest answer).

  26. Thanks for the great tips

  27. i wonder how come you can come upwith so good posts everytime.. Gr8 article.

  28. This is really good advice
    nice sharing
    Usability testing is a serious investment of time and resources for any team.
    Thanks a lot

  29. Very insightful from a web designer perspective.

  30. I love the pic of the girl with the hammer, hilarious.

  31. How can you be sure these mistakes are true? Is this backed by market research?

  32. Law Firm Websites,

    The first mistake has been proven true by about half a century of consumer surveys wherein consumers wherein what consumers say they want IS NOT what they want. The most notable example of this would be Coke’s research prior to launching “New Coke” – people claimed they liked new coke better, but that wasn’t at all how it turned out in the marketplace.

    The second rule has been proved by centuries of scientific research and law. In court, you’re not allowed to ask a leading question. And in psychology, your research is invalid if you tainted the subjects’ responses with leading questions. The same applies with user testing.

    The third mistake is somewhat of an extension or combination of the first two: don’t take what they say they want at face value. As Henry Ford said, if he’d listened to the customer, he would have designed a better horse and buggy!

    Mistakes 4 & 5 are really just encouragement or warnings against getting discouraged. Feel free to ignore them and get discouraged if you like ; )

    Lesson 6 is basic scientific knowledge. You have to limit your variables. In user testing, it also helps to keep the subject “fresh” by testing a few tasks. If you test too many tasks the user may get bored, tired, etc. You may also get too much prior-learning coming into play.

    Not sure if this is what qualifies as “market research” but the mistakes aren’t really presented as the distillation of some statistical analysis – they’re what one experienced User Testing professional passed onto her student and what he found to be most true and useful in her advice.

    - Jeff

  33. Great article. I’ve thought about placing an ad on craigslist to have a few people spend about 10 min testing my site. The article helped show me how to make the most of user testing.

  34. Very insightful from a web designer perspective.

  35. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  36. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  37. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  38. I think that the number 6 rule is the most important one. It happens all the time. people over optimize, over test and never get good results. It happens all the time.

  39. I think that the first rule is the most important, because recently I made a presentation, and it sucked, but no one give any suggestion, but said that this is great and bla bla.

  40. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  41. Never ask, “What do you think about this?”
    I usually do this.I will take care next time.

  42. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  43. Nice tutorial :) Great approach. I liked the point about design :)

  44. Thanks for posting this.

  45. Thanks for posting this.

  46. I will never ask question 1 about my web.The answer always make me feel bad….:P

  47. Very useful article, thanks for sharing these tips. I’ll start reading these things often as they are really helpful!

  48. Great tips. Especially number 6 – It’s so much easier to explain bite sized tests to the client.

  49. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titledTop 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  50. Completely disagree with 1 and 3. What he describes is a very basic usablity test but there is a great deal more that can be learned with a more creative approach which also solicits feedback. Basic task testing yields little more then that but limiting user testing to tasks alone completely eliminates the opportunity to reveal any of the Persuasive Issues that the E brothers write so eloquently about. Granted I would approach 1 and 3 differently they he describes but his approach leaves a lot on the table.

  51. Although I agree with you, conversion rate optimization is all about best practices first and then about intuition and niche.

    Lesson 6 is basic scientific knowledge. You have to limit your variables. In user testing, it also helps to keep the subject “fresh” by testing a few tasks. If you test too many tasks the user may get bored, tired, etc. You may also get too much prior-learning coming into play.

  52. I ran tests with usertesting.com at least 4 or 5 times and I got always something new and useful to improve face. I really like them. This encourages me; maybe I will be trying another test?

  53. Interesting points, thanks for sharing them with us.

  54. Lesson 6 is basic scientific knowledge. You have to limit your variables. In user testing, it also helps to keep the subject “fresh” by testing a few tasks.

  55. I agree that the it is really important how well and how easily the user can USE the website.

  56. [...] GrokDotCom has an excellent post on common mistakes made during usability testing, titled Top 6 User Testing Mistakes and How to Avoid Them. It provides a valuable outline for what not to do when creating your usability [...]

  57. phew what a relieving! Sometimes analytic tool can’t give best result..

  58. A nice blog.i think that this session will take a good, hard look at everything cool

  59. Lesson 6 is basic scientific knowledge. You have to limit your variables. In user testing, it also helps to keep the subject “fresh” by testing a few tasks.

  60. Completely disagree with 1 and 3. What he describes is a very basic usablity test but there is a great deal more that can be learned with a more creative approach which also solicits feedback. Basic task testing yields little more then that but limiting user testing to tasks alone completely eliminates the opportunity to reveal any of the Persuasive Issues that the E brothers write so eloquently about. Granted I would approach 1 and 3 differently they he describes but his approach leaves a lot on the table.

  61. I think that is important to ask “What do you think of this”. I feel that this question lets the person feel connected with the process. The key is after you ask that question, you must guide them to see it your way.

  62. All of the lessons are obviously valid under certain conditions. It’s important to be flexible with such lessons

  63. This is a really useful list, and I can understand fears from established UX professionals when something like usertesting crops up. However, it is arguably an opportunity rather than a threat: one can continue to trade with a ‘bespoke expert’ service and augment it with select user testing. (I have to declare a bias here as I have created another user testing service.)

    Notwithstanding, regardless of the tool or approach taken to user testing, the most important thing is that companies actually do some. And I am constantly amazed and appalled at how many simply skip it. It should be a fundamental part of any web project and not an ‘optional extra’ which is most often pushed down the priority list.

    Last thing I wanted to add is to address the elephant in the room: no amount of user testing (of any sort) is of any use unless the customer is prepared to actually pay attention to the feedback received and commit to understanding its significance and being prepared to act accordingly…

  64. @John Clark – thanks for your comments. We might say the same about Conversion Rate Optimization (our specialty), which should follow on the tail end of, or be done in tandem with, any user testing.

  65. [...] 6 User Testing Mistakes, And How To Avoid Them by Futurenow [...]

Add Your Comments

 

Print this Article
Share

Jeff is a Persuasion Architect, Web copywriter, blogger, and instructor of FutureNow's Persuasive Online Copywriting workshop. Follow Jeff Sexton on twitter

More articles from Jeff Sexton

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: