Measuring the Impact of a Civics Program for Middle Schoolers at the New York Historical

Program EvaluationImpact-Driven Strategy
Written By Cathy Sigmond

Client: New York Historical | Location: New York, NY | Funding: Institute of Museum and Library Services (IMLS)

 
 

We measured the impact of New York Historical’s Tang Academy for American Democracy, a program designed to introduce 6th graders to different models of democracy and sustain excitement about learning history through middle school.

OVERVIEW

In 2023, we partnered with New York Historical’s (NYH) to evaluate the Tang Academy for American Democracy (TAAD) program. TAAD aims to address 6th grade New York City learning standards by comparing the ancient Athenian model of democracy to democracy in the United States, as well as generate excitement about learning history.

TAAD had been in operation for three years (much of it taking place online during COVID-19), and NYH educators felt it was time to learn if the program was having their intended impact on students now that it was back to being held in person at the museum.

APPROACH

 We designed an evaluation with three phases that built on each other:

  • Phase 1 (Planning): We collaboratively developed a Logic Model with TAAD program staff to define measurable outcomes of the program. These outcomes served as guideposts to develop instruments for the following phases.

  • Phase 2 (Pilot Evaluation): We collected data about TAAD through two exploratory methods (observations and embedded assessments). These qualitative approaches helped us refine our instruments for the next phase.

  • Phase 3 (Full-Scale Evaluation): Using a quasi-experimental pre-post study design, we assessed student outcomes through standardized observations and an assessment embedded as a program activity to look for statistically significant student growth.

CLIENT TAKEAWAYS

TAAD students demonstrated significant growth in content knowledge about the two democratic models, skills (e.g., public speaking and collaboration), and social-emotional learning and civic engagement. Observations showed that students were most engaged when presented with opportunities to connect program content to their own lived experiences.

The evaluation also highlighted areas for increased alignment between the intended outcomes of the program and actual student outcomes, particularly those around object inquiry. In reflecting on the findings with NYH, we noted outcomes on the Logic Model that were less aligned with the program’s long term intended impact, and which might be replaced with ones that were more closely aligned.

NYH plans to make these changes to the Logic Model, as well as think about possible changes to the art making components of the program to support student creativity and interest in bringing their ideas about democracy and civic engagement back home to their communities.

Cathy Sigmond

Cathy brings many years of experience in education and experience design to her role as Head of Strategy at Kera Collective. 

Having previously worked in a variety of educational settings, Cathy is driven by her constant fascination and delight at how people make discoveries about the familiar and the unfamiliar. 

Cathy loves helping to shape experiences that spark curiosity and make a difference in people’s lives. She particularly enjoys the rapid, iterative nature of design-based research and the deep insights that come from qualitative research, especially on projects exploring interactions with the digital and built environments. 

Cathy shares her passion for experience design research widely and regularly guest lectures for graduate programs, including the Fashion Institute of Technology’s Exhibition and Experience Design Program and the Pratt Institute’s School of Information. Cathy served as the co-chair of the Museum Computer Network’s Human-Centered Design special interest group from 2018-2021.

Outside of work, you can usually find Cathy playing soccer, thrifting, or making her way through her large cookbook collection. 

Cathy’s favorite museum experiences are immersive; she will always vividly remember walking through the giant heart at the Franklin Institute, being surrounded by birds at the Peabody Essex Museum, and hearing centuries-old instruments come to life at the Museum of Musical Instruments. 

Previous

Creating a Shared Vision for Audience Impact at the Smithsonian’s Anacostia Community Museum
Next

Quasi-experimental Study of Youth Experiences at George Washington’s Mount Vernon