​Think Like a Respondent: Consumer Research’s Golden Rule

We’ve probably all seen examples of awful questionnaires or discussion guides. If you want more, there are plenty online—check out @MRXshame on Twitter for some hilarious ones. The crux of the issue is that we, as market researchers—client or supply side—have all been guilty of designing surveys that we would never want to complete ourselves. We conveniently forget or ignore how tedious those grid questions are, how annoying it is to answer the same question worded slightly differently multiple times, how impossible it is to remember something you bought six months ago, and how your attention span starts to wane after 10 or 15 minutes.

There are significant consequences of bad questionnaire writing, in the form of bad data and unreliable results—from straight-lining or random responses from the people who do finish your survey to chronically under-representing certain groups from people who drop out. For example, Quirk’s published a compelling study in February 2016 (“The impact of survey duration on completion rates among Millennial respondents”) which found that there’s a major drop-out inflection point among Millennial respondents after 15 minutes.
Education and training are critical to master good research design and quality, but even if we have a good foundation, we can still lose touch with the people who respond to and participate in our research. To that end, I want to offer a few simple suggestions that we can all start applying today to help us create research we would actually want to participate in ourselves.

1. Be someone else’s respondent.
Sign up for some online quantitative research panels or apps (e.g. e-Rewards, Field Agent, SurveyMini, The Pryz Manor, etc.) as a respondent. Always be honest—if they’re screening out market researchers, you can’t participate. But for the ones you can complete, you’ll get great ideas for what works, what to avoid, and how to make survey research more engaging. Nothing builds respondent empathy faster than taking a poorly-designed survey!

I would not recommend signing up for qualitative panels though. They should all have industry screen-outs and even if they don’t, the chance of messing up someone’s research in a qualitative setting with small base sizes is just too high.

2. Eat your own cooking.
Take your own survey. No, you might not be the target consumer, but you are a human. If filling out that complex matrix question drives you nuts—and you wrote it!—imagine how someone who doesn’t care nearly as much about your category/business will feel.

Look with alien eyes at that creative exercise you planned. Are the instructions clear? If you didn’t know what you know about your product/brand/category, would it make sense to you? How long will it really take to find all those images or complete that storytelling exercise?

3. Phone a friend.
Request peer-reviews of your questionnaires and discussion guides. If you’re on the client side, exchange surveys with colleagues for feedback, especially those outside your business unit/category if possible. On the supply side, you can also get feedback from co-workers, but just be mindful of confidentiality if you go outside the client team (i.e. use an in-market ad or package instead of the test one as stimuli, remove any proprietary client questions, etc.)

If you’re an independent consultant or don’t have ready access to colleagues for any other reason, strike a deal with a few trusted professional contacts to do a “feedback exchange” for questionnaires, guides, etc. where you review each other’s’ materials on a regular basis. The confidentiality caution applies here too.

4. Straight from the horse’s mouth.
There’s probably no better way to understand the real survey-taking experience than pre-testing it with consumers (i.e. not professional researchers.) There’s a range of ways to approach this—from very quick and informal all the way to an additional phase of research, depending on existing knowledge, business risk, and budget.

The most informal way to do this is to find people who fit the most basic criteria (e.g. pet-owners, restaurant-goers, detergent-buyers, vacation-planners, etc.) in your workplace or among friends and family and go through your screener, questionnaire or discussion guide with them. In this context, I think it’s most effective to administer it like a face to face interview where you read the questions out loud and mark their answers. You’ll get some instant feedback as you go (e.g. facial expressions, questions about the questions, etc.) and you can also ask for direct feedback too.

On the other end of the spectrum, if you’re planning a large-scale research project (multiple legs or geographies, a very high investment, or large potential business impact), doing a small qualitative phase up front to develop the questionnaire or guide can pay huge dividends. This also applies if you’re going to be researching a category/industry that’s relatively new to you and you don’t necessarily know all the right consumer language, response options, category attributes, etc.

--

Whether you’re learning to think like a respondent by actually being one, or getting feedback from a professional researcher or lay-person, we can all use these insights to make our research better—a little clearer, less complex, more engaging, and ultimately, higher quality.

---