Questions? (877) 643-7244
FutureNow Post
Tuesday, Mar. 4, 2008 at 3:04 pm

4 Q’s Your Web Analytics Can’t Answer

By Robert Gorell
March 4th, 2008

Our firm is very skeptical of surveys. They can annoy customers. The questions themselves tend to reflect the bias of the person or company asking them. At best, they offer directionally-correct information, which is often taken out of context and used to replace an old assumption with some shiny new (and equally dangerous) one.

The great thing about Web Analytics is that it allows you to see how people are using your site — without surveying them.

Our motto: “Believe what they do, not what they say they do.”

But there are things your Web Analytics can’t tell you, especially if you didn’t use personas to plan the site in the first place. And if you haven’t planned the site with personas, even believing “what they do” can be misleading.

That’s when a survey can help. When handled correctly, surveys can provide a good starting point for gathering qualitative data about the customer experience.

Not just any survey will do. You need the greatest survey questions ever:

  1. What is the purpose of your visit to our website today?
  2. Were you able to complete your task today?
  3. If you were not able to complete your task today, why not?
  4. If you did complete your task, what did you enjoy most about the site?*

Avinash Kaushik, the man who developed these powerful (and modestly-titled!) survey questions has teamed up with iPerceptions to create 4Q, a free tool that allows you to add this survey to your site.

Seeking Analytics Nirvana

It’s rather strange, isn’t it? Why would one of the World’s foremost experts in parsing quantitative Web Analytics data — the guy who wrote the book on it, actually — insist we get excited about fuzzy, qualitative data?

Let’s see what Avinash has to say about 4Q…

If you have read my book or my blog you are quite aware of the What and the Why issue. All the quantitative data you and I have from our web analytics tools is really good at helping us understanding the What happened.

Visits and Visitors, pages viewed, referrers, keywords, bounces, paths (!), campaigns, and so on and so forth. All critical data that helps you step up your game – improve your campaigns, fix pages, fire someone.

It cannot, no matter how much you torture the data, tell you Why something happened.

[...] We overlay our own opinions and experiences and preferences.

Unfortunately we are not our customers. In fact being as close to our companies as we are, it is quite likely that we are the worst possible people to empathize with our customers.

Sounds crazy, we know, but this whole “visitor empathy” concept might be the best thing that ever happened to your relationship with Web Analytics — let alone your relationship with actual customers.

In this YouTube presentation, Avinash explains how qulitative and quantitative data live in harmony:


. . .

*Fourth question added by iPerceptions.

[Disclosure: Future Now's Bryan Eisenberg is an adviser to iPerceptions.]

Add Your Comments

Comments (17)

  1. Robert,

    Many thanks for the piece.
    Just wanted to point out a typo in your * and disclaimer: iPerspective

    Thanks.
    Jonathan

  2. Jonathan,

    Thanks for writing in. Sorry about that!

  3. Robert,

    Thanks for the article. It would be great if you could provide a benchmark ratio (or a method for one) for how many survey completions it takes for the data to be considered actionable.

    Thanks,
    Luis

  4. Luis,

    Again, this is about getting qualitative data, not quantitative data. That means that if a single survey respondent — just one person — tells you about something that’s broken on your website that you were previously unaware of, their comment then becomes actionable. You can fix it based on what they say.

    If all the visitor tells you is that “Your site doesn’t have what I need,” that’s not actionable because the comment itself is of low quality. On the other hand, if the visitor says something more interesting — (“The product description for the Sony digital camera I was looking at doesn’t tell me how it’s any different from the older model, which you sell for half the price. Why’s it worth twice the price?”) — you then have something to take action on. Now you know you need better copy on at least two product pages because you’ve identified a question you weren’t addressing for that particular customer.

    With any type of qualitative data, you should use your judgment. Just because someone wrote in to tell you that you should change this or that with your website doesn’t necessarily mean that you should act on their advice. But you should always listen to them.

    Chances are that if someone’s taken the time to tell you about something your website could do better — or to thank you for what it’s already doing well — there are many others out there who feel the same.

    Look for trends in the types of things people are telling you. If certain requests or comments are often repeated, track how often such comments are made.

    Fortunately, your site’s current performance can be its own benchmark. What you must do is TEST any changes that are made based on the qualitative data you’re getting from customers. If things like conversion rate, average order value, form completions, and so forth, go up, then you’re on the right track.

    If you’ve made changes based on this qualitative data and the site’s performance has been stagnant, then either the changes you’ve made haven’t addressed the real problem.

    Keep listening to customers. Look for trends in what they say. See if it correlates with anything from your Web Analytics. And, most of all, TEST any new changes or assumptions you’re making.

    Does that help?

  5. I guess it comes down to voice-of-customer analytics is no different then behavioral analytics. They both have best practices.

    I dont think its about either qualitative or quantitative. I think its about the integration of the two. Qualitative data provides the “why” to behavioral’s “what”. Through integration qualitative data contextualizes the quantitative base, providing true user led recommendations.

    Surveying is a about methodology. It’s about invitation solicitation, the way you ask the questions, what questions you ask, and the way you analyze the data.

    Qualitative data is another dataset in the ever growing analytics toolbox.

  6. I have nothing against qualitative data but it is so much cheaper and easier to collect and analyze raw numbers.

  7. Robert,

    This is a very interesteing service, why do they provide it for free? Just for marketing exposure?

    I was wondering if you had a few sample URLs of sites that actually use the system so I can view it directly before going through the whole proces of signing up? I couldn’t find any samples on their website.

    Eric

  8. Robert,

    We are a leading MR firm specializing in gathering online user behavior.

    Relying on site visitors to accurately report on their behavior risks false reporting. We assign a realistic task to survey respondents and then track their clickstream in pursuit of that task. We ask them if they were successful and gather their impression of their experience. By correlating their opinion with their actual experience we are able to see a very clear picture of what really happened. We do this without putting any software on the site or requiring any client-side download.

    Marshall Harrison, CEO
    RelevantView
    http://www.relevantview.com

  9. My questions are the same as the ones Marketing Morgan asked:

    1. Why is this free? Just curious.
    2. Can we look at a few demos? Would love to get this for Chillibreeze.

  10. Nishi,

    1) 4Q is free! iPerceptions began with an enterprise level solution…and while we hope that a certain number of 4Q users will eventually upgrade to a full scale ‘webValidator Continuous Listening Solution’, we have no intention of ever charging for 4Q. It is what we call a ‘faith based intitiave’.
    As per my quote in our 4Q Press release:

    “With the web moving decidedly forward to meet the immediate needs of the end user, iPerceptions believes that all site owners have an intrinsic right to hear the voices of their customers,” said iPerceptions’ Vice President of Marketing, Jonathan Levitt. “4Q will allow the mass market to put their online customers front and center, which is exactly where they belong.”

    2) In terms of a demo. In the next few days, we’ll be putting a 4Q survey up on the 4Q website. In the meantime though, I would be happy to have someone walk you through a demo of the product. I would also recommend that you take a look at Avinash’s video (which is pretty much a demo). Go to YouTube and do a search on ‘Voice of Customer’.

    Thanks!
    Jonathan

  11. Thanks Jonathan!

    I have a question for all you readers. Let us say you are surfing a website and a survey window pops up – Would this affect your experience negatively? If you don’t want to take the survey, do you close the window and continue surfing? Would you click the back button because you don’t like pop ups? Would your answer depend on what the website was offering you?

    It is great to be able to get feedback from visitors but if this service is going to be free, lots of websites will use it. I am wondering if this will reduce response rates and even have a negative impact on the user experience.

    Sorry Jonathan (and Avinash). Just playing the devil’s advocate here.

  12. Hi Nishi,

    Just fielding this one for Jonathan. You’ve raised a great point, and it’s certainly a valid (and widely used) objection to doing website satisfaction surveys. That’s pretty much why we seek to obtain consent to participate upfront for a post-visit survey, rather than interrupting mid-session. We know that participating in a survey isn’t the world’s sexiest or most glamorous activity, but if the solicitation is clear and polite and the tone employed is one of brand-building, a solid proportion of visitors will jump in. And even if they are dis-inclined to jump in, just one click of the “no” button sends them back to the page they requested.

    Hope this helps!

  13. Actually, I visited iperceptions.com just now and I must say the window that “popped up” for the survey was pretty unobtrusive. Thanks!

  14. An issue that is being worked on is ensuring the initial “ok/no thanks” screen is presented in a way that maximizes visitor confidence that he or she has in fact reached the website desired, and not been maliciously redirected. Just having my logo added to 4Q’s initial screen might not be enough to keep NEW visitors from clicking away. Barring that, I’m very interested. Thanks 4Q/iPerceptions!

  15. I encountered a 4Q survey on the Union Privilege website (unionprivilege.org) and was impressed with it for several reasons:

    1. It clearly was branded to their site. There was no guessing that this was an advertising gimmick served up by someone else.
    2. The survey is short and sweet. Asking four questions is brilliant. I will not answer long surveys, but a four question one from a website I trust — yes I’ll do that.
    3. The actual survey itself was quick and easy.

    Seems like a great service to me!

  16. 4 Q’s Your Web Analytics Canâ��t Answer…

    The folks at Future Now are skeptical of surveys. They can annoy customers. The questions themselves tend to reflect the bias of the person or company asking them. At best, they offer directionally-correct information, which is often taken out of conte…

  17. I am interested in a survey of this kind, but 4Q does not offer enough customization for my needs. We have three separate audiences for our site, and the dropdown menu of tasks does not really address two of the audiences. Is there a paid survey out there that does essentially the same thing? I have looked at iPerceptions and Foresee Results, both are enterprise solutions. I would like a mid-tier solution that fits my budget.

    Thanks

Add Your Comments

 

Print this Article
Share

More articles from Robert Gorell

Marketing Optimization Blog
FREE Newsletter Sign-Up
send it once every: