How Many is “Many?”
There’s a lot to love about qualitative data. As we’ve written about before, its open-ended nature helps you understand people’s thoughts and experiences in their own words, revealing rich and profound insights in a way quantitative data just can’t.
Recently, we sent a report to a client with our analysis of some interview data. And our client responded and wanted to know more about how to interpret the language we used to “rank” trends in the data.
Here’s their [paraphrased] question:
“Can you give me a sense of what you mean by the terms “Many,” “Most,” “About Half,” “Several,” and “A Few?” Not looking for hard and fast percentages, but an indication of what would imply more people versus fewer. Is there a mental ranking (most to least) that you have in your head when you write these terms?”
I closed my eyes, channeled my inner Elyse Myers and thought, “Great question, I would love to tell you.”
Embrace the mess
Let’s back up for a second. When we’re analyzing data from quantitative methods like surveys, which have closed-ended questions like rating scales or multiple choice, we use hard numbers and percentages. It’s easy to see that, for example, this many people chose Option A, this many chose Option B, and this many chose Option C on a multiple choice question. Reporting the data is relatively neat and tidy (although attaching meaning to numerical results is not always straightforward).
Qualitative data, on the other hand, is full of description, detail, and nuance—and with that richness often comes a lot of messiness.
When analyzing qualitative data, our end goal is always to provide a sense of how much or how little an idea or trend came up within the whole sample. Sometimes, it’s very clear—people either say “yes, I liked this” or “no, I didn’t like this,” and there were no overlapping reasons why. But most of the time, it’s less straightforward and more nuanced. We have to interpret what people say based on many factors, including:
Tone (did they really mean the words they said, or does their tone imply otherwise?)
Hesitation (are they just taking a second to think, or are they being cautious not to offend?)
Groupthink (are they just agreeing with everyone else in their interview or focus group because it’s easier?)
Holding opposing ideas (if someone says two opposing things in the same sentence, should I consider the first idea or last idea they said to be their final answer, are they unsure what they think, or are their feelings really a little bit of A and a little bit of B?)
… you can see how it’s not always simple!
Could have been 10, could have been 14
This openness to interpretation is what makes qualitative data exciting. It’s also why when reporting qualitative data, we generally steer clear of numbers. Instead, we use descriptive terms that reflect our categorization of the trends because it’s important to “keep qualitative data first and foremost qualitative.” This quote from influential evaluator Michael Quinn Patton encapsulates this idea well (I even had it printed next to my desk for a while!):
“Ironically, the adjectives “most,” “many,” “some,” or “a few” can be more accurate than a precise number. It’s common to have responses that could be included or omitted, thus changing the number. When I code 12 of 20 saying something, I’m confident reporting that “many” said that. Could have been 10, or could have been 14, depending on the coding. But it definitely was many.”
If you take one thing away, let it be this: with qualitative data, the exact number of people who said what is not as important as considering trends relative to one another. In other words, you should focus on the ideas that are rising to the top and falling to the bottom rather than on parsing out the precise number of people who fall into a given category.
More of an art than a science
Maybe all this is making you sigh in relief (“Phew, it’s open to interpretation!). Or maybe it is stressing you out (“Is anything real?”). Or maybe you don’t know how to feel.
I get it. So to try and bring some semblance of order to the chaos, here is a graphic defining some of the most common descriptors we use when writing up qualitative data and our general rules of thumb for defining them. This by no means an exhaustive list—rather, it’s a starting point. Save it for the next time you are writing up trends emerging from qualitative data or reading a qualitative report!
P.S. This post focuses on how to read qualitative data after it’s been analyzed and interpreted. Wondering how many people you should talk to when gathering qualitative data? Read this one to find out why the answer will always be it depends (and why that is totally okay!).