I recently attended the inaugural West Coast Quirk’s Event in Irvine, California, a market research industry conference focused on “big ideas, real-world solutions”. Looking back, the unofficial theme of the presentations I saw seemed to be: let’s get closer!
 
This played out in multiple ways: get closer to reality with new ways of collecting data, including in-the-moment research, get closer to consumers/customers with co-creation and empathy in research, and finally, get closer to seeing the whole picture with mixed methods and data integration.
 
Here are a few more specifics and examples from the conference to bring these themes to life.
 
In-the-moment research ensures authenticity in data collection

• Dave Decelle, Netflix, shared his experiences with using in-the-moment video diaries to learn about when and what people are watching video, without recall bias. To make it even more true to life, the team started with their own internal customer data to identify actual viewing behavior trends and then created customized schedules to push surveys out in prime viewing periods.

• Layla Shea, Upwords, reinforced the need for in-the-moment research by highlighting the differences between stated intentions and reality in her presentation, “The Road to Ruin in Paved with Stated Intentions”. Humans are notoriously inaccurate with recalling memories as well as predicting their own future behavior. She gave examples of using digital journaling to conduct in-the-moment and in-context online qualitative research.

• Kelley Styring, Insight Farm, shared how she uses “selfie ethnography” and digital missions as applications of behavioral science principles in research. In addition to conducting the research, these techniques can also be applied as a screening tool. For example, she might ask participants in a multi-phase research project to refer a friend and then “audition” that person via a digital ethnography assignment.

 
Co-creation and shared experiences compress timelines and foster empathy

• David Lundahl, InsightsNow, talked about the trend of corporations moving from linear to “spiral” innovation approaches, which is requiring a parallel shift in research approaches. Some of their new approaches include: testing all aspects of a proposition at once, relying more on behavioral (vs. claimed) insights, and the use of wireframes/prototypes earlier in the process. They provided a case study of a 3-day iterative design sprint with concept co-creation, prototype-matching, and a larger-base validation step, which saved the client about six months versus their traditional process.

• Catherine Cooper, EmpathyWise, shared a compelling case study from a financial services client seeking solutions for the under-banked, applying the design thinking framework. In the Empathize stage, they did one-on-one interviews with customers using metaphor elicitation and other projective techniques. The resulting customer stories gave deeper understanding and new inspiration for the client team, leading to the development of a totally new business model for that target audience.

• Debbie Balch, Elevated Insights partnered with Gail Tomiak, USAA to share the story of how they brought USAA customers to the company’s executives in an interactive, conference-style one day event. The executive group interacted with 100 current members in a combination of face-to-face and virtual conversations. This event brought the USAA members to life in a new, richer way for participating company leaders, replacing facts and figures with faces and stories. Executives walked away inspired, energized and full of new ideas, solutions and opportunities.

 
Mixed methods and data integration provide holistic insights

• Candice Rab and Becky Wu, Luth Research, shared an example of their consumer Pathways to Purchase methodology which combines multiple research modalities to truly capture the entire process. In addition to traditional qualitative and quantitative research, they also have access to digital passive tracking data (across devices) to understand search, social and other online behavior. This approach also integrates geo-fencing and geo-tracking capabilities to capture brick and mortar behavior, allowing true omni-channel insights.

• Evencia Leite, TrendSource, Inc., shared an insightful case study about their work on the Pokémon brand that focused on multi-source integration. The client had a very broad set of objectives, ranging from key driver of retail execution all the way to new product opportunities, so the agency designed a multi-prong learning plan to provide holistic insights, including both primary and secondary quantitative data (e.g. retail audits, panel surveys) as well as qualitative approaches (e.g. customer intercepts, shop-alongs).

​

• Tyler Kettle, IBM, talked about their need to move from large scale projects that meet the needs of the few to continuous, customizable research. To accomplish this, they initially developed an online research community, which has since evolved into their “Always-on Intelligence Platform” which encompasses: social monitors, insight platforms, bi-weekly pulse surveys, and deep-dive custom research, as needed.

We’ve probably all seen examples of awful questionnaires or discussion guides. If you want more, there are plenty online—check out @MRXshame on Twitter for some hilarious ones. The crux of the issue is that we, as market researchers—client or supply side—have all been guilty of designing surveys that we would never want to complete ourselves. We conveniently forget or ignore how tedious those grid questions are, how annoying it is to answer the same question worded slightly differently multiple times, how impossible it is to remember something you bought six months ago, and how your attention span starts to wane after 10 or 15 minutes.

There are significant consequences of bad questionnaire writing, in the form of bad data and unreliable results—from straight-lining or random responses from the people who do finish your survey to chronically under-representing certain groups from people who drop out. For example, Quirk’s published a compelling study in February 2016 (“The impact of survey duration on completion rates among Millennial respondents”) which found that there’s a major drop-out inflection point among Millennial respondents after 15 minutes.
Education and training are critical to master good research design and quality, but even if we have a good foundation, we can still lose touch with the people who respond to and participate in our research. To that end, I want to offer a few simple suggestions that we can all start applying today to help us create research we would actually want to participate in ourselves.

1. Be someone else’s respondent.
Sign up for some online quantitative research panels or apps (e.g. e-Rewards, Field Agent, SurveyMini, The Pryz Manor, etc.) as a respondent. Always be honest—if they’re screening out market researchers, you can’t participate. But for the ones you can complete, you’ll get great ideas for what works, what to avoid, and how to make survey research more engaging. Nothing builds respondent empathy faster than taking a poorly-designed survey!

I would not recommend signing up for qualitative panels though. They should all have industry screen-outs and even if they don’t, the chance of messing up someone’s research in a qualitative setting with small base sizes is just too high.

2. Eat your own cooking.
Take your own survey. No, you might not be the target consumer, but you are a human. If filling out that complex matrix question drives you nuts—and you wrote it!—imagine how someone who doesn’t care nearly as much about your category/business will feel.

Look with alien eyes at that creative exercise you planned. Are the instructions clear? If you didn’t know what you know about your product/brand/category, would it make sense to you? How long will it really take to find all those images or complete that storytelling exercise?

3. Phone a friend.
Request peer-reviews of your questionnaires and discussion guides. If you’re on the client side, exchange surveys with colleagues for feedback, especially those outside your business unit/category if possible. On the supply side, you can also get feedback from co-workers, but just be mindful of confidentiality if you go outside the client team (i.e. use an in-market ad or package instead of the test one as stimuli, remove any proprietary client questions, etc.)

If you’re an independent consultant or don’t have ready access to colleagues for any other reason, strike a deal with a few trusted professional contacts to do a “feedback exchange” for questionnaires, guides, etc. where you review each other’s’ materials on a regular basis. The confidentiality caution applies here too.

4. Straight from the horse’s mouth.
There’s probably no better way to understand the real survey-taking experience than pre-testing it with consumers (i.e. not professional researchers.) There’s a range of ways to approach this—from very quick and informal all the way to an additional phase of research, depending on existing knowledge, business risk, and budget.

The most informal way to do this is to find people who fit the most basic criteria (e.g. pet-owners, restaurant-goers, detergent-buyers, vacation-planners, etc.) in your workplace or among friends and family and go through your screener, questionnaire or discussion guide with them. In this context, I think it’s most effective to administer it like a face to face interview where you read the questions out loud and mark their answers. You’ll get some instant feedback as you go (e.g. facial expressions, questions about the questions, etc.) and you can also ask for direct feedback too.

On the other end of the spectrum, if you’re planning a large-scale research project (multiple legs or geographies, a very high investment, or large potential business impact), doing a small qualitative phase up front to develop the questionnaire or guide can pay huge dividends. This also applies if you’re going to be researching a category/industry that’s relatively new to you and you don’t necessarily know all the right consumer language, response options, category attributes, etc.

--

Whether you’re learning to think like a respondent by actually being one, or getting feedback from a professional researcher or lay-person, we can all use these insights to make our research better—a little clearer, less complex, more engaging, and ultimately, higher quality.

Talking with physicians in Frankfort, visiting spas in Hong Kong, in-home interviews in Moscow, shopping for skin care in Shanghai, and watching men shave in Paris—these are a few of the amazing opportunities I’ve gotten to learn about consumers all over the world. Conducting consumer and market research outside your home country brings with it plenty of interesting experiences and stories, new and rich learning, but also comes with plenty of potential pitfalls. Here are five key areas to focus on when planning international qualitative or quantitative research.
 
1. Lay the Groundwork
When planning primary research in another country, it’s helpful to first get some understanding of the unique market dynamics, key competitors, consumer habits and practices, etc. If you don’t already have access to secondary research in that market, some smart internet searches can go a long way. It will give you good context for creating meaningful research questions as well as helping you better understand local consumer responses.
 
For example, before conducting a quantitative survey on workplaces in Australia, I checked out blogs for ex-pats working there—it was a great, quick way to get some compare and contrast insights about unique workplace culture and habits in that country. In another case, before doing consumer interviews about skin care services in Hong Kong, I visited lots of local spa websites to check out the service menus, benefit language, etc. If you’re doing internet research for a country with a different primary language than your own, Google Translate is a lifesaver (not word perfect translation, but you get the gist).
 
2. Supplier Partnerships
As with any research project, finding the right research partner is key for success. When conducting research internationally, you generally have a choice between using a large, multi-national full service supplier or going straight to a locally-based, often more specialty, vendor. In general, I would recommend using a multi-national firm if you’re planning research in more than one country for the benefits of a single point-of-contact for coordination and logistics. Depending on the country, those firms then either have local offices for execution or contracts with local research companies. If you’re focusing only on one international market (or region), going straight to a locally-based company can get you closer to local experts who know their market best and is often more cost-effective.
 
When conducting international research, I rely heavily on my local research execution partner to offer guidance on market-appropriate research approaches and techniques as well as recruiting and incentive best practices. I would also recommend giving and getting detailed plans, timelines, scope, etc. in writing up front to minimize potential misunderstandings from language differences or general expectations. When conducting qualitative research, if you’re working through a global firm, make sure you also have a locally-based contact that will physically be present during the research and is familiar with all the details, as tracking down someone in another country/time zone can be difficult if a question or issue comes up during research.
 
3. Research Logistics
When scheduling research internationally, be sure to carefully check local holiday and events schedules—ideally directly with a local as there may be ‘blackout’ dates for research that aren’t necessarily obvious based only on a calendar. For example, August in Europe is best avoided as many people are on holiday that month or, while Chinese New Year is officially one day on the calendar, celebrations extend far beyond that.
 
Take guidance from the local recruiter, but when planning qualitative research in other countries, typically plan for multiple extra recruits per group, especially in large urban areas. Traffic, weather and even local attitudes toward time/punctuality can all impact respondent arrival time.
 
While I’ve certainly tagged along on many in-home and shop-a-long interviews in other countries, it’s important to hire a skilled and experienced local moderator to lead interviews and focus groups. Not only because of potential language differences, but also for the cultural understanding and nuance—both in the interview and the ability to share contextualized insights after.
 
If you will be physically traveling for research in another country, it’s critical to pre-plan all logistical details and not take anything for granted. Make sure you know what resources will and will not be available at the research site (e.g. copier, projector, wireless internet). If you’re doing in-home or shop-a-long research, will you need to hire a driver or will public transportation be more appropriate?  
 
 
4. Translation
Conducting research in a country with a different primary language than your own presents many challenges. Many local research companies will have bi-lingual staff, especially those with ties to the large multi-national suppliers, which is very helpful, but these people are usually not professional translators so make sure to secure those resources separately.
 
Live, simultaneous translation of qualitative research is a very specialized skill so if you need a translator for that, try to get recommendations from others or at a minimum, make sure that the research translator specializes and is experienced specifically in that field. Also make certain that the local focus group facility is set up with the proper tech equipment for simultaneous translation, particularly if some attendees will be listening in the local language and others to the translation.
 
Translation of questionnaires and research stimuli (e.g. concepts, package copy, advertisements) is another consideration. Most global research firms offer translation of screeners and questionnaires, but many will not translate research stimuli. Check with your research vendor to be sure, but also consider hiring a third party company that specializes in translation for market research.
 
With any written translation, be sure to request a “back translation” as well. Back translation is a translation from the translated foreign language back into your own language. This can often help catch mistranslations, especially for local idioms and culturally-specific meanings or terminology. For example, “anti-aging” skin care in English might get translated to a nonsense term in the local language that literally back translates to “against being old”. Explaining the intent, giving examples, and talking to locally-based bi-lingual contacts can help find the right terms to substitute.

5. Response Scales and Cross-Country Comparisons
​Many side-by-side studies have been conducted and academic papers written on the subject of cultural bias in research scales, so I will not attempt to address all the nuances of that here, but I do want to just mention the need to be aware of variances in how different cultures use response scales in questionnaires. For example, in Asia, scale order is typically in reverse of scales used for research in North America because Asian consumers are usually more likely to give a more positive or affirmative response vs. their North American counterparts. So, a response scale used in South Korea might list agree options from ‘Strongly Disagree’ down to ‘Strongly Agree’ while the same question in the US version of the study should list ‘Strongly Agree’ to ‘Strongly Disagree’.
 
These cultural dynamics also impact how questions are asked in a qualitative setting to be as non-biasing as possible, so be sure to involve a locally-based researcher, ideally the moderator, in writing the discussion guide and planning any creative activities and exercises within interviews.
 
If you’re conducting quantitative research across multiple countries and want to be able to compare the results, there are a few things you can do. First, consult an expert (within a multi-national research firm or a research consultant with global experience) for guidance on tailoring your questionnaire by market. This could involve re-ordering sections or individual questions, using different scales or response order, or even customizing the instructions you give for filling out the questionnaire. Also make sure your scales themselves are as non-biasing as possible; consider using a numerical scale (e.g. rate from 1-10) vs. a qualitative scale (e.g. rate from ‘excellent’ to ‘poor’).
 
Probably the most effective way of understanding a country’s specific results relative to another country is to use benchmarks and databases. If your company conducts lots of multi-national research frequently, you might be able to construct your own internal databases and benchmarks for comparison. Otherwise, many larger global research vendors maintain robust results databases by country. For example, if your concept gets a top two box purchase intent score of 40% in the UK and 50% in Japan, you can compare those results to norms for the category and country for more insight. You may learn, for example, that a 40% score in the UK is actually in the top 20% of that database, while a 50% score for the same category in Japan is actually only in the top 40% of that database.
 
-
 
Many lessons in effective global research can only come from experience, so talk with colleagues and research vendors and consultants who’ve been there before if you’re just getting started with international market research. The most important thing is just not to assume that everything is the same in other markets as in your own home country—do your homework, ask lots of questions, and leverage the experience of others while you build your own. Doing consumer research in other countries is an eye-opening and fascinating experience. It’s a unique chance to learn about other cultures, appreciate our differences and similarities, and gain fresh insights for our business.