Using Survey Data Starts with Careful Planning

Evaluation & Research Design
Written By Stephanie Downey

Useful survey data doesn’t magically appear at the end of an evaluation study. It starts at its inception with a question you ask yourself: 

Why are we collecting surveys in the first place?

It's tempting to jump straight to what questions you will ask visitors— a few rating scales, some check-boxes, a comment box. The survey questions seem fine. You might collect plenty of data. SurveyMonkey might show you bar graphs, a pie chart, and a word cloud. It feels like you are “doing evaluation.” 

But a survey built without a clear purpose can produce data no one uses. That's not a data problem. It's a planning problem.

Before you survey visitors, take time for careful planning by asking yourself these questions:

  • Why do we want to collect data? How will the data inform our work? Why is that important to us right now? What decisions can it help us make? Can we really change the way we do things based on the results?

  • Who needs this data? Is it important to a program manager? Program facilitator? Funder?  Who needs the data will influence what questions you ask.

  • How will we collect the data? Who do we want to survey?  How do we ensure we capture their responses?  Do we have the capacity for that?

  • Who will analyze and interpret the data? Does that person have the time and skill to do so?

  • With whom and how will we share the results? Will we write a report? How can we make the results relevant to others on our team? 

Careful planning with a clear purpose leads to a survey instrument with intentional questions and meaningful results. 

Case Study: Careful Planning in Action

We worked with a science museum to survey visitors during its annual outdoor science festival. Before writing a single survey question, we had an in-depth conversation with the museum team to understand why they wanted to do a study. For this example, I'll focus on just the first planning question — why they wanted to collect data in the first place — though the same kind of thinking shaped their answers to the rest.

The museum team had a number of related problems they wanted data to help them solve. They said the festival pulled in a large crowd every year with no advertising or targeted outreach, meaning the museum didn't actually know what was bringing people in, or keeping others away. In answering the very first planning question — why do we want to collect data — the museum landed on a list of specific things they needed to know:

Awareness: Did people realize the festival ran for many days, with dozens of events, and that the museum itself produced it?

Motivation: Were people coming for the reasons the festival was designed around, or did most people see it as just a families-with-kids event? What themes might people be interested in for future festivals?

Reach: Were they reaching people who didn't already think of themselves as "science people" — and which populations were missing?

Welcome: They had heard second hand that some people in their nearby community said they didn't feel welcome visiting the area. Was the event itself accessible and welcoming?

Distinctiveness: Was the festival’s mix of science and art an attracting force for visitors? They needed real answers, not assumptions, to decide where to invest next.

Impact: Was the festival meeting its goal of helping visitors think about science in new ways and relate it to their everyday lives?

Each issue or problem the museum raised was then translated into specific items on a short exit survey. You can see on this annotated copy of the survey how the museum’s problems map directly to one or more survey questions. Results of this survey were incredibly concrete and actionable because of the time spent planning. Careful planning doesn't just produce useful data; it produces answers you can actually act on.

Stephanie Downey

Stephanie brings more than two decades of research and evaluation experience to her position as owner and director of Kera Collective.  

She is driven by her lifelong passions for education and equity and informed by her training as an anthropologist.

Stephanie takes pleasure in working closely with museums and other informal learning organizations to help them leverage their strengths to make a difference in the lives of people and the wider world.

Stephanie has a national presence in the museum field, regularly presenting at conferences like that of the American Alliance of Museum and the National Art Education Association, as well as teaching and guest lecturing in universities such as Bank Street College and Teachers College at Columbia University. She was on the board of directors of the Museum Education Roundtable, serving as treasurer, from 2016 until 2021. 

When not working, you can find Stephanie in the kitchen trying new recipes, cheering on her children in their various activities, and hiking trails in the Hudson River Valley.

Stephanie’s favorite museum is the Lower East Side Tenement Museum because it combines many of her favorite things: an authentic and immersive historical setting, stories of ordinary people in extraordinary circumstances, and gritty New York City.

Next

Follow Your Curiosity