Smart Survey Design - Don’t Bore Your Respondents (and Get Better Results!)

My initial title of this article was going to be How to Write a Better Survey, but I nearly fell asleep while typing it.  Instead I decided to get straight to the point and walk through a few of the proper ways to write survey questions to ensure you get the most out of your respondents.

  1. Try to use simple, easy to understand wording.  When working with clients to design the surveys, they tend to want to use technical words or phrasing that, while many of the respondents who are most familiar with their products will understand, can sometimes be difficult for the average respondent to grasp. Ask questions as you would in a conversation.  For example, instead of asking “What are the primary reasons you rated that attribute with a 9?” you might say, “Why did you say that?” or “Why did you rate a 9?”.   Keeping your questions short and to the point will help your respondents more clearly focus on their responses.
     
  2. When writing open ended questions, ask for a specific number of desired responses.  Research shows when you ask respondents to give you a list, rather than a general question, they are less likely to give a single answer or response.  For example, when asking “Why do you purchase from that supplier?”  You might get some substantial responses, you will likely receive more one-word answers, or IDKs.  By rephrasing the question such as “What are the top two reasons you purchase from that supplier?”, respondents are much more likely to respond, and with more than one-word answers.
     
  3. Whatever methodology, try not to ask what they cannot answer.   I work with clients all the time who have a survey they would like administered online and via phone.  While there are many analytical issues we could delve into (comparing online vs phone data, sample sizes, etc.) I am going to concentrate on the questions themselves and the primary issue of: will this work both on the phone and online?

    One of our best examples of this comes from a recent study we worked on.   The client was adamant about asking the respondent to allocate 100 points over five different attributes based on what is most important to them.

Now that you have taken some time to think about your supplier.  Here are five areas that may influence which supplier your company uses.  Please allocate 100 points across these items giving more points to those items that are more important to you when selecting your wholesaler or distributor.  Give less points to an item that is less important to you.

You can tell already that this is a very lengthy question, and we haven’t even made it to the answer choices. 

(Phone interviewer instructions: Read all items first before having the respondent give points.  Repeat the items if necessary)

  • Pricing
  • Ease of using programs and processes
  • Consumer marketing such as national ad campaigns & store branding
  • Being an owner or association member
  • Merchandising programs and support

You may have noticed that many of the attributes listed are phrases.  In an online environment, it is easy for the respondent to see the attributes and compare. However, over the phone this is a very complex process which often requires the interviewer to repeat answers, adding to the length of the survey, and more importantly adding to the frustration of the respondent.  For a question such as this, if the client is adamant about keeping it, we would recommend a maximum of four choices and keeping them to one or two words each.

This is just one example, but the overall point is to keep all questions in mind when developing your survey, and if they will work for the appropriate methodogy(ies).

4. Keep it under 10 minutes. It can be difficult to keep someone’s attention, even more so today.  People want their information fast and don’t always have a lot of time (or want to devote a lot of time), to a survey.  I know some of you are saying, “But we use panels and the respondents get incentives!” or “We are paying them $20!”  That is wonderful, and it gets those respondents to begin the survey, but regardless, they will still only stay focused for so long.

If your survey is over 10 minutes, here are some things you can do to shorten it:

  • Remove any questions that will not meet the objectives, or are just there because “you/the client would like to know that too”. Are there too many objectives?  Maybe you need two surveys.
  • Are there any questions you already know the answer to? Remove them.
  • Are there any questions that you don’t know the answer to, but the answers would not be used or acted upon?  Remove them.
  • Think about rotating questions (i.e. displaying a set of questions to half of your sample, and displaying another set of questions to the remaining sample).  Yes, you get half the data, but depending on how much data you are collecting, for some questions you may not need as large a number of completes.

The whole point of surveys is to get good, reliable, quality data with true answers.  Keeping your survey simple and short is the best way to achieve this.

Written by: Grant Gries, ggries@thefarnsworthgroup.com