There comes a time when designing a research program that one must ponder the question that has haunted researchers since the beginning of our profession: Which should I do first, qualitative research or quantitative research?
The truth is, as researchers, there isn’t always an easy answer. Regardless, we try to approach business questions agnostic to the methodology. We are committed to helping clients discover what is significant, where we need to dig, and where we need to explore further.
Begin at the End
It may sound strange, but when starting a new project, it helps to think about “the end” of the project first. Envisioning what you want to do with the data post-analysis is the key to any research design. What is our end goal for socializing and activating future insights from this work within the organization? What deadlines and business objectives are we working towards? Who are the stakeholders that will use these results, what roles are they in, and what is most important to them?
A successful mixed-methods approach takes all of these end goals into account to determine the best course of action. Even if teams come to the table with a preferred methodology in mind, what separates a typical vendor from a true insights partner is that we listen closely to the long-term business needs first and then suggest an approach, which may or may not be exactly what the original team had in mind. Many clients don’t realize how beneficial a mixed-methods project can be—in many cases, you can execute two projects in one, making better use of internal and external resources.
It can be tempting to get wrapped up in a philosophical discussion about which should come first—qual or quant, chicken or egg. Our approach at Conifer is to get down to business: what is the business question, and what is the context of that question within the company’s overall business strategy?
Put another way, sequencing must consider the business need from the onset of scoping every project. Instead of just throwing a productized research approach at the business problem, we like to take into account your team's existing knowledge, business goals, timelines, and desired outcomes to help you make informed choices about the appropriate sequencing methods. That way, we’re determining the sequencing not based solely on the process, but on your desired outcomes.
Qual Before Quant
Qual Before Quant makes the most sense when we are at the early stages of the learning curve, or even when exploring an entirely new territory altogether. In these situations, we may not even know how consumers talk about a topic, think about a topic, what they find appealing or challenging, or why they do things a certain way. Qualitative data helps us to figure out what ideas and topics should be addressed with quantitative methods. Without this foundational data, research efforts are lacking a yardstick or metric. If you don’t know what you are measuring or what your measurements are, you can’t write a great survey.
Furthermore, leading with qual also allows us to learn the “natural” consumer language used when talking about a product or category so we can be sure to use that language in our quant survey, rather than defaulting to our habitually-formed internal jargon.
In some cases, by first conducting open-ended research, we get a clearer picture of what consumers include in their consideration set — giving us a more accurate view of how users engage with the category. By using qualitative methods to observe all the different ways the product is being used, we are able to uncover the sources of delight and frustration along the way, meaningful workarounds, and even how competitors may be addressing similar questions. Whatever the need, rich qualitative methods are first in line to provide us with many, if not all, of the possibilities which we can then take to Quant and measure with large samples.
An example of the “Qual Before Quant” approach at Conifer can be found in a recent study we conducted in the financial services space. Our goal was to understand how emerging consumer cohorts (specifically Gen-Z and Millennials) think about and engage with their money. Our in-depth ethnographic research and interviews provided us with a solid understanding of their assumptions, fears, hopes, and expectations — all of which we were later able to confirm directly in a follow-up quant survey. In Phase 1, we heard loud and clear in our in-depth interviews that the main modes of interaction are currently mediated via automated FinTech solutions… yet Phase 2 quantitative work revealed there is still a strong desire for personal interaction, confirming that “the ability to connect with a real person” was still very much a top criteria when choosing a financial company, revealing important boundaries on technology.
Quant at the back end of a project also makes more sense when we have too many of “something” — say, a list of user needs, new ideas for our product pipeline, or new feature options for a product, and we want to narrow and size down the choices based on consumer-centric criteria, like their needs and interests. To continue with the example of our research with Gen-Z and Millennials, our in-depth qualitative research helped our client to generate a few dozen or so ideas to pursue in order to make better inroads with these audiences. Our team was then able to take these ideas to hundreds of respondents in a quantitative survey — applying a highly effective Max/Diff analysis method to clearly separate the winning ideas from the rest.
Quant Before Qual
While advanced qualitative research is known for its depth, there are also many examples of quantitative research that deliver a deeper understanding of a space, measuring new areas, or audiences, with greater focus and rigor. There will always be situations when we need to dive in, feet first, with quant to get our numbers and figures straight to better understand the questions being asked.
More often than not, teams are not starting a research study from scratch; there can often be a wealth of existing knowledge in teams with a well-established area of expertise. However, we may find that, in past studies, the data has not been cut in the best way to consider our specific question (i.e., the right audience wasn’t targeted, or our questions have fundamentally changed due to new players in the market). These moments will call for a fresh survey instrument or a new sample strategy to get us started.
Take the case of “Attitudes & Usage” studies (also known as A&U, or U&A) — one of the foundational sources of information for many companies. An A&U study aims to provide an understanding of a specific product category and its key players to gain a deep competitive landscape view. A&U studies give us a clear picture of who uses what/when/where/how, as well as all the underlying perceptions, likes and dislikes, delighters, and pain points.
A very common and useful next step with an A&U study is to conduct a segmentation analysis to identify the distinct groups of consumers with their unique needs and expectations. A good segmentation study can guide a wide range of decisions, from targeting to positioning to communication strategies and innovation. A typical segmentation study often starts in the quant space as it requires large datasets and robust analysis with such techniques as k-means or hierarchical clustering. But what makes segmentation especially powerful and widely adopted across organizations is what happens next: following it with rich qualitative research methods to breathe life into segments in the form of “personas.”
Qualitative personas following quant segmentation help “elevate” the quantitatively-defined audience groups by creating a clear qualitative picture of who they are as individuals: their lifestyle (or, culture in the B2B world), values, and needs. At Conifer, we have no shortage of anecdotes from our clients pointing to the impressive power of life-sized cardboard cut-outs of “real people” paired with posters detailing their real stories decorating their office walls, acting as constant reminders of who their customers are and the importance of putting them at the forefront of key decisions.
Quant once more takes the lead when it comes to helping teams with prioritization via innovation team workflows. Innovation teams are always in ideation or iteration mode, so it is not uncommon for them to come to us with multiple active ideas or concept spaces being explored or incubated in tandem, asking which idea or concept warrants investing further resources via live consumer testing.
In a recent project collaboration, our team began with over 200 new-to-the-world technology use cases that a subset of our client’s business was working on. After synthesizing the 200 use cases into 40, we started with a large global quant survey to get a broad, gen-pop read on how each of these 40 would stack up across key global markets. We used Max Diff analysis to identify which use cases rose to the top in terms of being “more valuable.” With this information in mind, we next conducted a digital ethnography and in-depth interviews to delve deeper into 8 use cases ranging from high, low, and average rankings. Why not just explore the top performers? Because here, getting an understanding of why specific use cases were more or less appealing to consumers was more important than understanding only the top list. Being able to concentrate on ideas that quantitative research showed as less promising can help us pinpoint the specific attributes or mental models that drove favorability.
Why Choose? Qual and Quant Studies in Parallel
“What is good for the goose is good for the gander” — while this might apply to birds, it doesn’t always apply to market research. There’s an entire world of needs that exists beyond any given research study driven by the business. This means that sometimes the methodology choice is dictated more by external forces and timing needs. For example, an upcoming product launch requires early, if imperfect answers, a prototype or concept that needs to be tested isn’t going to be ready quite on time. Running quant and qual in parallel is sometimes a very real necessity — and while it has its tradeoffs, it can bring its own value and benefits too. Having both quantitative and qualitative teams tuned in and working alongside each other as a cohesive unit can also yield its own wonderful synergies.
In a recent study, when time was of the essence to complete multiple research streams before an end-of-quarter deadline, we leveraged our wealth of past research to kick-start our qual and quant research simultaneously. Simultaneous to fielding a global psychographic study, we conducted in-depth interviews, probing deeper into many of the areas we were also covering with our survey. When both pieces of research came to an end, we were able to weave together a clear picture of our client’s target consumer, backed by not only hard data, but also by stories and images that showed who the consumer was, what they valued in life and the category, and how the new product we were researching might fit in their existing system.
Iterative Studies - Round and round we go!
Here’s when things get really interesting — and also a little dizzying! Sometimes our client partnerships are so iterative and long-standing that whether the chicken or egg comes first is completely irrelevant! It is not uncommon to be in a constant cycle of qual and quant research, especially for innovation teams. The life of a product, end-to-end, involves an ongoing chain of analytics and insights needed to bring it to fruition and maintain its relevance. For example, we may start with a quant segmentation to identify our core target, dive deeper into their needs and white-space areas through qualitative methods, brainstorm new ideas using design research skills, run a quant survey to sift through how those ideas are received, evolve a select few with further qual research, test them in a follow-up concept test, do another round of qual prototype exploration to fine-tune details, and, if you thought we were done yet… leverage more advanced quant to finalize features, pricing, make prototypes, then test them in a centralized location test (CLT) or in-home test (IHUT), develop the packaging and then test that, and then develop the ads and test those… and on and on until at least we have our finished product in-market! (....phew, that was a lot!)
We find that when you’re in the business of knowledge and curiosity, one piece of research always leads to another... especially when we uncover unexpected findings that beg an immediate follow-up.
So, which sequencing methodology will work best for your project? As the wise Obi-Wan Kenobi once said, “You will find that many of the truths we cling to greatly depend on your own point of view.” Simply put, there isn’t a clear-cut answer, and one size does not fit all. The “right” approach is one that takes into consideration the business outcomes you’re aiming to achieve through research and works backward from there.