Summative Evaluation: Is It Worth It?
At the Visitor Studies Association (VSA) conference in July 2024, I co-presented with two museum exhibition practitioners about questioning the status quo of exhibition evaluation practices. We each approached the session with questions and skepticism about traditional summative evaluation for exhibitions. The big question I explored was: Are summative evaluations worth the cost (money and time)?
The Status Quo, Traditional, Summative Evaluation
Traditionally, summative evaluation examines the successes and shortcomings of a program, exhibition, or project against its stated goals for users. For exhibition evaluations:
What they explore: Summative evaluation often examines the extent to which visitors achieved the desired outcomes or learning objectives. Outcomes or learning objectives can be classified in many categories, including cognitive, behavioral, or affective. In plain language, whatever the exhibition design team intended for visitors to think, do, or feel, that is what you examine!
How / what kinds of data: We typically collect firsthand accounts from visitors about their experience with the exhibition, either via interviews or surveys. To complement firsthand accounts, summative evaluations sometimes include timing and tracking or other forms of observation to document what visitors do while in the exhibition (e.g., time spent, components used, conversations had, behaviors, etc.). Observations are powerful because—paired with visitors’ first hand accounts—they can help evaluators explain visitors’ meaning making experience in the exhibition. For example, an interview reveals that a visitor understood Idea A but not Idea B or C. When you look at where they spent time and attention in the exhibition, it indicates they skipped over a section where Idea B and C were emphasized heavily.
When it happens: Summative evaluation happens after an exhibition is “complete” and open to the public. We typically suggest summative evaluation happen a few months after an exhibition opening to allow the exhibition team to resolve any issues that naturally emerge after install prior to evaluation. Data about visitors’ exhibition experiences can be collected immediately after or sometimes weeks after the experience.
My Experience with Summative Evaluation
As context, I entered the museum evaluation field in 2008. VSA’s Evaluator Competencies had just been published as part of work to professionalize the field, which includes competencies related to understanding and being able to measure outcomes of informal learning environments. In this same year, the National Science Foundation (NSF) developed a Framework for Evaluating Impacts of Informal Science Education (ISE) Projects. An impetus for this framework is to create more standardization in summative evaluation to allow for “cross-project portfolio reviews, which they had previously found to be widely varied.” That is, NSF and the framework designers wanted to be able to compare evaluations to identify trends across projects.
Most of my experiences have been in this post-“professionalized” world of museum evaluation. My project portfolio at the time contained many summative exhibition evaluations, including NSF-funded evaluations for the Fort Worth Museum of Science and History’s CSI: The Experience and the Science Museum of Minnesota’s Wild Music, to name two projects. As we entered the 2010s, my work included less summative exhibition evaluations but many summative program evaluations, including evaluation for the Museum of the City of New York’s Traveling Through Time program and a very large impact study of single-visit K-12 art museum programs for the Museum Education division of the National Art Education Association and Association of Art Museum Directors. And while I was excited to do these types of big, complicated, systematic studies at the time, I have now begun to wonder…
Is Summative Evaluation Worth It?
As a study that comes “at the end,” summative evaluation has little effect on the thing being evaluated (as compared to front-end and formative evaluation). This is particularly true for summative evaluations of exhibitions. Temporary exhibitions have a limited exhibition lifespan, and if they do travel, there may not be adequate budget to change them before traveling. “Permanent” or core exhibitions have greater potential for remediation than traveling exhibitions, but this is again contingent on having a budget for remediation. Summative evaluation for programming provides greater opportunity for change, in my mind, since programming generally costs much less to produce than exhibitions.
Yet, whether evaluating an exhibition or program, I still wonder about whether the benefits outweigh the costs considering:
Money: Summative evaluations cost anywhere from about $25,000 to hundreds of thousands of dollars depending on the number of methods, sample size, etc. Since they have little impact on the product they are evaluating, might that money be better spent elsewhere?
Time: Summative evaluations take a lot of time for the evaluators, museum practitioners, and most notably evaluation participants, such as adult visitors in a museum or students on field trips. Evaluation participants are sometimes paid or given a token of appreciation for their time (particularly today as opposed to 20 years ago), but I think it is important to remember what an important commodity time is. In conducting summative evaluation, are museum practitioners using their and research participants' time wisely?
Growth and Learning: Ultimately, as an evaluator, I believe in the benefit of evaluation as a learning tool. However, my concern is that the benefit of summative evaluation lies primarily via the kind of cross-project review or literature review advocated by NSF, which I don’t think is happening widely. So, does summative evaluation generate the growth and learning to really affect the direction of museums, policies, and actions in the informal learning community?
Reflection
The VSA session that prompted this post generated a lot of conversation. I leave you with two questions to ponder:
Is a summative evaluation (for exhibitions, programs, projects, etc.) worth it in your opinion? Why or why not?
What type of evaluation, summative or otherwise, feels worth it to you? Why?
I’d love to know what you think!
