Learning and Evaluation/Archive/Learning modules/2Impact Evaluation

Jump to page

Part 1. Overview: Program Evaluation Basics Overview; A Note about Words; What is a Program?; Understanding Programs; What is Program Evaluation?; Why is evaluation important?
Part 2. Stages of Evaluation: The Cycle and Stages of Evaluation; The Tasks of Each Evaluation Stage
Part 3. Types of Evaluation: Types of Evaluation; Needs Assessment; Feasibility Assessment; Monitoring; Context Evaluation; Formative/Process Evaluation; Summative Evaluation; Impact Evaluation
Feedback

Program Evaluation Basics Menu

Impact Evaluation

Stages Involved

Completion & Feedback		Identification

Implementation		Design

Key question:

How much difference did it make?

Quantifies the contributions of a program toward a long-term outcome.
Reveals whether a program or a change in a program contributed to long-term outcomes (i.e., changes to the social, economic, civic, and/or environmental conditions to which the program was targeted).
Requires a comparison of indicators; either an examination of program participants over time or a more careful examination of your program participants against a separate control or comparison group.
Provides comparable data about programs to inform decisions about continuing, expanding, or reducing funding based on cost of attaining impact.

Impact evaluations tend to be the most complex in design. Not only does the evaluation of impact carefully examine how much a program contributes to one or many long-term effects, it often incorporates findings from other evaluation types (i.e. context, process, etc.) applied along the planning and implementation path.

Impact is determined when one can:

compare the condition(s) of a group before and after a program occurs and state that the program has changed the condition(s) of the group;
measure the value added by a program compared to either an existing program or alternative practices.

Many questions can be answered through impact evaluations, including: To what extent does a program increase the number of retained editors? What is the value of transforming participants into retained volunteer editors through a program? How much more or less impactful is a program compared to other programs that also aim to increase retained volunteer editors?

Evaluation in Action: The Impact Evaluation

The chapter created the new Wiki Loves Monuments program in small and rural towns because they wanted to make an impact on editor retention. They suspected that a more focused theme might help participants to feel more connected to their country and might lead to an increase in dedicated, long-term editors.

An impact evaluation measures the extent to which a program effected change in terms of program outcomes. An important component of impact evaluations is to compare a program's long-term outcomes to either the general population or to another program. For example, one might compare editor retention of participants from Wiki Loves Monuments in rural and small towns to the general rate of editor retention on Wikimedia projects. Editor retention from the newly modified program could also be compared to editor retention from a traditional program.

The chapter conducts the evaluation and finds that this program increases editor retention by 2% compared to new users to Commons, but seems to have about the same impact on editor retention as traditional Wiki Loves Monuments. With this information, they are able to make a decision as to whether they want to continue with the traditional model or adopt the new one (or do both).