Learning and Evaluation/Efficiency, effectiveness and impact
Part of the Wikimedia Resource Center
Learning and Evaluation
This page aims at introducing the reader to three basic concepts of program evaluation: efficiency, effectiveness and impact. It aims at defining these terms while giving some real-life examples in the Wikimedia context. After reading this page you will have a better understanding of each of these terms and you will also know what they mean in the context of doing programmatic work within the Wikimedia movement.
Defining program efficiency, effectiveness, and impact
[edit]A great starting point for talking about program evaluation is to get a better understanding of the concepts of "efficiency", "effectiveness" and "impact". Once we all agree on how we define and use these terms, we will share a common language for everything else that follows. Because all three are things that we will want to measure later on.
Program Efficiency relates to the cost of producing products or services relative to other programs or to some ideal process.
Cost Effectiveness relates to an analysis of the costs – money, people, time, materials, etc. – that are expended as part of a program in comparison to either their benefits of their effectiveness (Boulmetis / Dutwin 2011, p. 5). What does this mean in our context? Let's consider two Wikipedia Editing Workshops executed by grantees A and B. Both grantees staged their workshops as one-day events and both had 30 participants attending their events. Grantee A asked 5 Wikipedians to train the participants how to edit Wikipedia, whereas grantee B needed 10 Wikipedians to do the same task. Now, let's just assume that both workshops have the same outcome. Every single participant's level of knowledge about how to create a user account, how to start a new article, how to work in sandboxes and how to upload a picture to Wikimedia Commons increased significantly, and at the end of the day, every participant was able to contribute to Wikipedia and to Wikimedia Commons (now, I guess you're getting it: this is a hypothetical example; but although this might never occur in reality, let's just take this as a way to explain the phenomenon of "efficiency"). Well, determining the efficiency in those two cases is easy: grantee A's workshop was twice as efficient as grantee B's workshop. Grantee A only needed half the number of people to achieve the same result as grantee B, whatever the reason for this might have been. Maybe grantee A selected Wikipedians who already had some experience in teaching newcomers how to edit. Or the workshop that grantee A organized had a better agenda that enabled a smaller number of trainers to cover the same amount of content. Let's not get further into the details here and move on to the next term instead.
Program effectiveness relates to the level by which the activities of a program produce the desired effect. Let's consider a grantee C who receives money for creating online training materials for Wikipedia newcomers. Those training materials introduce new editors to the basic concepts of how to contribute. As the materials are freely available online and many people can access them, grantee C has achieved a high level of efficiency. Why's that? Grantee C can reach a much larger audience with her online training than grantees A and B, so the materials prove to be more cost-efficient (let's just assume that C's investment was not bigger than that of grantee B). But are those online materials that grantee C created also as effective as the trainings that grantees A and B executed? As it turned out (oh, you're getting it again, right? This is also a hypothetical example; of course there are online trainings that can be as efficient as in-person trainings. I'm just making this up to explain the concept of "effectiveness"), after measuring the results, grantee C finds out that her online course was not as effective as she'd hoped. Only 10% of the participants who took the online course learned the basics of how to edit Wikipedia. Thus C's program was not as effective as A's and B's program. So, when you look at the effectiveness of your program, you are asking whether the activities did what they were supposed to do. Therefore, a program's effectiveness "is measured in terms of substantive changes in knowledge, attitudes, or skills on the part of the program's clients" (Boulmetis / Dutwin 2011, p. 6). Now, let's move on to the last term.
Program impact is the extent to which long-term and sustained changes occur in a target population (Boulmetis / Dutwin 2011, p. 7). Now we're really getting into what doing programs is all about. Let's consider that one of our strategic goals is to make more people contribute to Wikipedia and also to improve Wikipedia's coverage and quality of content. Thus, the impact of our programs can be measured by looking into how many people actually edit Wikipedia after going through one of our programs and by looking at how much these people's work improved Wikipedia. To explain this further, let's get back to our two prior examples. Half a year after executing their programs, grantees A, B and C decide to measure their programs' impact. They look at the participants who went through their workshops (A, B) or used their online materials (C) and count the number of people who actively started editing. They also measure the amount of content that those people contributed to Wikipedia. As it turns out (and this is still hypothetical, I almost don't dare to mention it again), out of the 60 people who attended grantees A's and B's workshops, 20 became active Wikipedians, each improving more than 50 articles. Grantee C had a different outcome: More than 1,000 people took her online course, and only 10% of those became active Wikipedians, each of them also improving more than 50 articles. That's more than 100 new Wikipedians and more than 500 articles improved for C's program, whereas grantees A's and B's effort resulted in 20 new Wikipedians who together improved more than 100 articles. Although C's program was less effective, it had a bigger long-term impact on Wikipedia.
The definition of 'program efficiency' in the original version of this article relates to cost effectiveness NOT efficiency. The issue is apparent when you analyse the difference in the post between 'program efficiency' and 'program effectiveness.' The only real difference is that 'efficiency' compares effectiveness against the cost. The concept referred to as 'program efficiency' in this model comes after Effectiveness. You need to understand the effectiveness of the program before you can measure 'program efficiency.' It is, as I argued above, cost effectiveness' not efficiency.
The literature defining efficiency in such terms (e.g. Cugelman & Otero 2010) is a minority view (see for example, http://betterevaluation.org/evaluation-options/value_for_money). Efficiency relates to the energy used in processes and the minimisation of waste inherent in such processes (check any dictionary). The definition of program efficiency needs, as most such definitions do, to reflect this common understanding of efficiency or the use of the term will lead to confusion and misunderstandings. Program efficiency has nothing to do with the benefits of the activity or its effectiveness. An efficient program is one in which the cost per product or service is low relative to some other program, or relative to an ideal. Measurements of program efficiency relate to the cost in relation to products or services NOT to benefits or outcomes.
Some of the confusion arises from conflating outputs with outcomes. While some of the literature equates outputs with "measurable change towards a goal" most of the literature limits outputs to products and services. So while, program efficiency does relate to outputs, outputs are not the results of activity (outcomes). Outputs are the products and services provided by the program.
Secondly there are at least two competing definitions of impact. The definition used here for 'impact' is not inherently different from those for vision, goal, outcome etc. It make little sense to use impact for such concepts. The competing definition, and the one I prefer, relates to the consequences of the program, both intended and unintended.