Your Market Insights Program Is Only As Good As the Questions You Ask

Making the case for semi-structured qualitative research

Data-Driven Decision Making

Most professionals have heard this term in the workplace. What does it mean, though? According to Northeastern University, “data-driven decision making (or DDDM) is the process of making organizational decisions based on actual data rather than intuition or observation alone. Every industry today aims to be data-driven.”

It’s true: before making sizable investments in product design, prototyping, manufacturing, and sales channels, companies should seek gather and interpret data which measures the market potential for new and improved products.

So what’s the problem? Why is the quest for data-driven decision making often shortsighted?

Skipping Straight to the Surveys

Surveys are a powerful tool to gather a large number of responses to specific, focused questions. Surveys are useful when you need to track consistent measures over time, such as customer support metrics. Surveys are also useful when you are making final decisions about a product or marketing program.

Do you know what surveys aren’t great at? Telling you if your baseline assumptions were flawed. Telling you if you’re looking in the wrong direction. Are you surveying the wrong group of people? Are you asking the wrong questions? You might not figure these things out until after you’ve run the survey and got confusing results.

Related to every new product development and market launch, there are “unknown unknowns” which are impossible to predict or know where and how to look for them. By definition, then, you cannot expect to uncover the most powerful insights through surveys because you would need to anticipate that issue to write survey questions to explore it.

Trying to Survey Segments Who Are Unlikely to Respond

Some population segments are easier to survey than others. Participation and response rates vary based on a variety of demographic and psychographic factors. Powerful, busy executives are highly unlikely to respond to your survey. Individuals with limited broadband or wireless access are also less likely to respond to your survey. Parents with more than one young child are also, in my experience, hard to engage for more than a few minutes without interruption.

Asking Too Many Questions

Survey fatigue is real. No matter whether you’re collecting data using online, mobile, or retail intercept surveys, you should expect low participation and high abandonment rates. Consumers are overwhelmed by the number of surveys and by the number of questions in each survey.

In the attention economy, the drop-off point occurs more quickly than ever. In an unpaid study, it could be as few as five questions. In a paid study, it may be more but still not that many.

To ask the same number of questions you could ask ten years ago, you may need to create multiple, shorter questionnaires.

Qualitative Market Research Insights Are a Necessary First Step

When you start with qualitative research, you will probably uncover some surprises that challenge your assumptions.

Discovering the Questions You Didn’t Think to Ask

Sometimes the surprises are bad, but they save you from costly mistakes. You might learn that your prototype needs to be reworked because there’s an issue in your design that’s non-negotiable for 80% of your target segment. You might learn that your initial concept is too expensive to produce, given the price points of your competitors and the product margins required by your distribution and retail partners.

Sometimes the surprises are good. You might learn that a different consumer segment than the one you expected is interested in your product. They’re also willing to pay for it at a higher price than you thought. You adapt your branding, product positioning, and retail channel strategies accordingly.

Sometimes, you knew what topic to ask about but you didn’t know what question to pose. In one case, my client wanted to understand the role gift-giving would play on sales. We were both surprised, however, to learn who was giving the gifts and what factors influenced their choice of gift.

Sometimes, you find something interesting which is neither objectively good nor bad. These are the insights you want to turn into testable hypotheses, which you follow up with quantitative research tools. You should use what you’ve learned to write more focused, impactful questions so you’re measuring the right things.

The Temptation to Undervalue Qualitative Research

Great market insights aren’t cheap. Leading consumer products companies spend a fortune on their market research programs, and they start with qualitative research tools. Teams with smaller budgets may try to save money by jumping straight into creating surveys. By now, though, you realize this is often a mistake. Qualitative research can give you much deeper insight into how and why consumers buy products.

How to Balance Quality, Cost, and Urgency

If you’re operating in a particularly fast-moving market like consumer electronics, SaaS, or toys, there often isn’t time for rigorous research with large sample sizes. In the weeks it takes to design and implement a research program — let alone analyze and report results — market dynamics may change completely.

Nevertheless, when you’re launching an important new product, you cannot afford to skip the exploratory qualitative research.

Here’s a pragmatic plan:

• Hire a highly experienced researcher who you trust can make the trade-offs between “urgency” and “rigor.”
• Start with a small set of customer interviews. Create a semi-structured interview guide with open-ended questions. Give your market researcher the flexibility to probe for more insight or to skip questions, based on where the customer is leading the conversation.
• Review the qualitative insights report. Decide which insights have implications that require immediate action, such as a significant design flaw. Decide which merit further exploration with additional qualitative research or validation using quantitative market research tools.

Readers, what are some surprises you’ve uncovered in customer interviews, focus groups, and other qualitative tools?