Is a Survey Really What You Need?
“Why don’t we just do a survey?” If I’ve heard it once, I’ve heard it a thousand times. Surveys are the first instinct for many museums seeking to learn about their audiences (and I’m sure the same is true across many industries). Some people use the term “survey” as shorthand for any form of asking questions to visitors—but surveys have a specific definition. Surveys, also known as questionnaires, present a set of standardized, (mostly) closed-ended questions (e.g., multiple choice, checkbox, or rating scales) to respondents and yield quantitative data that can be analyzed statistically.
Don’t get me wrong, I love a good survey, and they have many benefits. BUT, they are not always the best fit depending on what you want to know about your audiences. So I’m breaking down the Who, What, Where, When, and Why you should consider before deciding that surveys are the best method for your next audience research or evaluation project. These questions are not exhaustive, but offer a starting point for decision-making. Let’s start with Why, because it’s essential to first make sure your method fits your purpose.
Why are you doing this research?
Are you interested in learning about your audience’s background and preferences? Surveys can be a great way to learn more about your audience: Are they majority first-time or repeat visitors? What brought them to the museum? Did they feel welcomed during their visit? Multiple choice and rating scales are two standardized survey question formats that work quite well for exploring these ideas. With a large enough sample you can begin to confidently draw conclusions about your audience as a whole through statistical analysis, if that is important to your research goals.
Are you exploring a new approach in an exhibition or program? Surveys are often not the best fit for exploratory research, where nuance and detail are important to understand the whole story. This is typically true of front-end and formative studies, where qualitative methods are a better fit because ideas are still in the early stages.
Are you engaging with a sensitive or challenging subject? Surveys might be a good fit, but it depends. Surveys can provide a level of anonymity that encourages audiences to respond honestly to questions about sensitive subjects that they may not want to verbalize in a one-on-one interview or focus group where anonymity isn’t possible. However, surveys may also mask the tone of audiences’ responses, and they do not allow for asking follow-up questions that clarify or elaborate.
Who are you most interested in hearing from?
Is your audience small (a cohort of teachers in a professional development program) or large (all walk-in visitors)? Quantitative methods, like surveys, are most effective when used with larger audiences because the large sample size allows for statistical analysis that will allow you to confidently make inferences about the broader population. Or, you can use segmentation to look at differences among your audiences. Qualitative methods, such as case studies, interviews, or focus groups, may be a better fit when you want to deeply explore unique identities or perspectives that might be lost when data is aggregated into a large group.
Is your audience likely to respond accurately and authentically to a survey? Make sure you are being responsive to the preferences and needs of your specific audience. There are always important cultural considerations that could impact your survey design and analysis or that suggest a different method might work better. For example, this Pew Research Study found that Hispanic communities were more likely to respond with courtesy bias in surveys (overly positive responses) or give responses they perceived as more socially desirable.
What is your budget?
Is your budget low? Time spent on analysis is often a large part of any research budget. If you have a short, simple survey of mostly multiple-choice or rating-scale questions, simple analysis can happen quickly (ultimately costing you less money) because the standardized results can be quickly calculated. Conversely, it usually takes longer to analyze qualitative data (such as interviews or focus groups) because they are not standardized and require skill to code and interpret.
If your survey is long, tackles complex ideas or topics, or you want to understand relationships among variables, then survey design and analysis will require more skill and thus a higher budget.
Where is your audience located?
Is your audience geographically dispersed? Surveys could be a good fit because they can be easily administered online at any time, no matter where your audience is located (as long as they have internet access). While interviews and focus groups can also be conducted remotely, they have the added logistical hurdle of scheduling a time for the interviewer and participant(s) to meet. When your audience is located across time zones, scheduling gets increasingly complex.
Is your audience onsite at your institution? Great! Surveys could be a good fit, but data collection context is critical. For example, if you leave a stack of paper surveys at a ticket counter or use an unattended kiosk for visitors to fill out surveys voluntarily, you likely won’t end up with reliable data that truly reflects your visitation. Conversely, using a data collector to intercept visitors and ask them survey questions aloud will often improve sample and data quality, but this also requires additional time and/or resources.
When do you need the results?
Do you need results fast? Because surveys contain mostly standardized, closed-ended questions, you can quickly get a snapshot of your results. If your survey is hosted online through SurveyMonkey or Google Forms, these platforms will even automatically create summary charts you can use to get a sense of the results at a glance. That said, we always recommend closely reviewing the raw data to identify and address data issues, such as duplicate or invalid responses, that could throw off your results.
Can you wait a little bit longer for your results? Write-in responses (such as “Other, please specify”) allow some flexibility for people to offer responses in their own words when the available options don’t suffice. This can provide a bit of a bridge between strictly closed-ended quantitative options and the openness of qualitative data to welcome new or unexpected responses. However, keep in mind that written responses will take additional time to analyze because they cannot be quickly tallied or averaged like quantitative data.
So, as you approach your next audience research and evaluation project, pause and consider whether a survey is truly the right method for your Who, What, Where, When, and Why?
