Getting Big Ideas Out of Small Numbers

“Is that really going to be enough people?”

When the topic of user research comes up with a new client, they're often surprised by the small number of users we want to speak to. It’s important that designers and others involved in the design process understand research methodologies and can articulate the value we get from speaking to a small number of users.

Quantitative research involves large sample sizes of participants (think thousands) and is concerned with answering questions about how much, how often, and how many. Quantitative studies can be used to understand how often people spend doing certain activity, the size of a potential market, typical demographics, and user preferences. This research usually takes the form of surveys, web analytics, and other machine-gathered information. Quantitative research is good at helping us understand more about what we already think we know. Quantitative research isn't good at uncovering motivations, goals, or getting a high-level understanding of the people that will use a product or service.

User research at a call center.

Qualitative research on the other hand usually involves a small sample size (think dozens) and is concerned with understanding how people behave, how they think about certain activities, and what factors affect their behavior and thought patterns. This research takes the form of individual interviews in the context or setting where the product would be used (e.g. at the desk, in the car, etc.). The context or setting is important so we can observe what people do instead of what they say they do. Qualitative research is really good at helping us understand things we don’t already know.

At Cooper, we make use of both kinds of research. Most of our projects start with qualitative user research in order to build empathy and gain insight into the context in which the product or service will be used. This is particularly important when we're designing for unfamiliar and complex domains, but is useful for consumer products as well. This is very different from usability or user testing, which typically takes place after design in order to observe people using the product.

We do qualitative design research because qualitative techniques are better at revealing information that designers need: the how and why behind people's behaviors. These high-level insights help us frame design solutions around what really matters to our users and enables us to think of new and novel ways to address problems that may not be explicitly defined. Since qualitative research relies on small sample sizes, research can be performed and synthesized in a relatively short period of time at lower cost. Qualitative techniques also allow design researchers (at Cooper: interaction and visual designers) to be nimble and change the approach to get the information they need. After qualitative research we develop personas that can be used to vet our design ideas and solutions.

A common concern for clients and stakeholders is that the small number of users necessary for a qualitative study doesn’t seem “scientific”. Qualitative research is backed by behavioral and cognitive science but does not need large numbers of participants to achieve meaningful results. Because of this we need not be concerned with statistical significance; qualitative research is not statistically significant and that’s ok. It’s simply important to speak with enough participants to uncover and explore high-level goals.

  Number of Participants Types of questions Methods Results
Quantitative Large ("thousands") Closed (demographics, preference, likes, dislikes, etc.) Surveys, A/B testing Charts, specific results ("30% of participants are under 25")
Qualitative Small ("dozens") Open-ended (behavior, pain-points, goals, motivations) Observation, interviews Personas

It’s also important to remember that qualitative design research is not about asking users what they want. It should be no surprise that people have a difficult time articulating what they want, and often express desires in terms of something they already have. In order to give ourselves room to invent new and innovative products and services it’s important to focus on high-level motivations and goals. If it was as easy as asking people what they wanted and building it, we'd be surrounded by great products and services. It’s not easy to understand the motivations underlying behaviors, and that’s where ethnography can help.

Understanding the fundamentals of quantitative and qualitative research helps us be better researchers, communicate better with clients, and defend research methodology and results to stakeholders and others that don’t understand research methodologies. A lack of understanding of research methods can get in the way of valuable design research, leading to misunderstood or misinterpreted results.

If you have a specific question to answer, use quantitative research methods such as surveys or A/B testing. Quantitative research can help you understand who is currently using a product or service and can help target which participants to speak with for qualitative research. If you have open-ended questions, and want to understand the the pain-points, goals, and motivations of participants in order to create new and innovative products or services, use qualitative methods like interviews and contextual observation. There is no substitute for talking to people to get these answers.

For further reading reference Kim Goodwin's book Designing for the Digital Age. (Chapter 4, p.55)


Greg Schuler

Learn more

Subscribe to our mailing list and stay up to date on our latest thought leadership and learning opportunities.

Connect with us

Want to know how we can help your company drive real business progress?

Let’s talk

Stay up to date on our latest thought leadership and learning opportunities