It wasn’t so long ago that when I would see the words “measurable outcomes” on a grant proposal, I would experience a wave of nausea and anxiety. I would be required, the grant stated, to prove to the prospective funder that our programs and activities had created a better life for those who touched clay, and for the rest of the city - and maybe the rest of humanity. Wow! A tall order. Not only were we to prove how a small clay airplane can save the world, but save it by what percent over last year, and how airplanes versus cups, for instance, can have a significant impact on a community.
I have a personal bias against lying. I especially dislike lying to funders who could hold the artmaking future of our activities in their unpredictable hands. So, I decided that I’d better set to the task at hand and figure out how to evaluate joy, how to measure creativity, and how to quantify that “I get it!” moment that makes weeks of hard work worth the effort.
First, I had to look at what kinds of evaluations we were already doing and why we were doing them. Most of the evaluations that we were and are doing, we are doing for ourselves. (I’ve since learned that “ourselves” is called an internal audience.) These exercises include wrap-up sessions after fundraising events, surveys with our on-site participants, meetings with our community arts teachers and partners, annual performance reviews with staff, tracking direct mail fundraising appeals for their effectiveness, and other means of trying to find out if and how Baltimore Clayworks was making a difference. Furthermore, most of these activities were preceded by planning sessions. If little or none of this information was of interest to funders, why were we taking the time and making the effort to meet, write, and ask all of these questions? We were trying to strengthen our programs and trying to maximize the use of our resources — both very sound reasons to conduct evaluations.
In February 2000, Clayworks organized a weekend retreat to identify the elements of success as well as the challenges for our community arts program as it grows. We grappled with the core questions — how do you know when the artistic experience is authentic? How can you measure joy? The consensus was “you know it when you see it.”
Enter Maryland Association for Nonprofit Organizations (MANO) in early April with a two-day Program Planning and Evaluation workshop led by two researchers from Brandeis University. Out of my desire for Clayworks’ scant professional development hours and dollars to be used wisely, I queried the presenters by phone before enrolling. “Will I learn how to measure children’s creativity and joy in this workshop? Will you talk about the arts as well as human services?” When I was assured that my evaluation questions would be addressed, I wrote the check and blocked out the two days.
Eureka! What a revelation! Formal, methodological evaluation, I learned, was not just for rounding out grant proposals and following up funding, it was for strengthening programs and allocating resources! And what’s more, at Clayworks, we were already doing many evaluative procedures and doing them properly. But what I learned that was of enormous value, is that by instituting a down-to-earth, commonsense, understandable framework for evaluation, we could achieve a level of consistency and determine our programs’ effectiveness with far more accuracy than before.
This framework (the “Theory-Based Logic Model Evaluation”) is based on a series of “If-Then” assumptions that are tested by conducting activities (programs — making those clay airplanes, cups, etc.) that have early, interim, and final benchmarks that are either met or not. The meeting of benchmarks is the measurable outcome, and the outcomes determine the ultimate impact of the program. The logic model can be used for both internal (ourselves) audiences and an external (funders) audiences, but the assumptions, or what exactly you are trying to evaluate, can be tweaked for the particular audience. Sounds simple?
It may be, but as we begin to use this system we’re finding some challenges. First, it takes time.
We are holding staff training sessions of 3-5 hours on the method; that may not seem like many hours, but to pull half the staff together for training can be complicated. The programs’ planning sessions must include getting all of the people involved to discuss and agree upon the assumptions as well as the program activities. We must find evaluators to attend the programs at the beginning, middle, and end, and they need to be able to collect data in multiple ways. If the evaluators are program staff or volunteers, they must also be trained. Finally, the program team needs to be able to analyze the data, and make decisions concerning the programs’ effectiveness and ultimate impact. At Clayworks, we are gradually moving into logic model evaluations. As part of staff training, we are putting some existing programs and events through the model, and using the exercise as a way of familiarizing ourselves with the language and thinking. We are also choosing to measure only those things we deem critical, not just what it would be nice to know about.
I don’t foresee a day in the near future where all of Clayworks programs will be evaluated by using logic models, but we’re making a start. In those areas where it is harder to measure success, and where gains and positive steps in creative activities are more incremental, and where the measurable outcomes are more ephemeral, we’re moving to institute logic model evaluation. Can we measure joy? Yes we can. And by knowing where and how to look for it, how to document what we see and hear, and how to communicate that, Baltimore Clayworks can create opportunities for more joy more often.