Learning and Evaluation/Setting goals and measures

Media about the SMART Goals process: Original FDC

or check out these presentations on setting SMART objectives:

Designing SMART Proposals – Wikimania Learning Day 2015
Designing SMART Proposals and Annual Plans – Wikimedia Conference Learning Day 2015

Goals, objectives, and the chain of outcomes in your program theory

Your theory of change, is a theoretical pathway which articulates the action steps that:

Link your mission and programming activities toward change through logical cause and effect relationships
Allow for the specification of program outputs and participant outcomes on the way to the impact you are trying to make
Focuses on key outcomes that are specific, measurable, attainable, realistic, and time-bound

A logic model framework can help you to review and map out your program's theory of change.

From there, it is up to you to prioritize goals to focus on, the relevant outcomes, and what measures and targets will be most important to tell the story.

Program Goals

Broad and future-oriented statement(s) of the desired condition(s), goal statements:
Identify the specific target group(s) for the goal,
Provide the “what” as distinct from the “how,” and
Target when the goal will be achieved or come about in important and measurable ways.

Examples of Common Wikimedia Program Goals include:

Building and engaging community
Increasing accuracy of information
Increasing quality of contributions
Increasing awareness of Wikimedia projects
Increasing buy-in for the free knowledge/open knowledge/culture movements
Increasing contributions to the projects
Increasing diversity of contributions and content
Increasing diversity of contributors
Increasing positive perceptions about Wikimedia projects
Increasing reader satisfaction
Increasing the usefulness, usability, and use of contributions
Increasing the use and access to projects
Increasing peoples editing/contributing skills
Increasing volunteer motivation and commitment
Increasing respect for the projects (i.e. higher education acceptance)
Making contributing fun
Making contributing easier
Recruiting new editors/contributors
Retaining existing editors/contributors

As an example, program activities and inputs and direct products (outputs) for the goal: “Recruiting new editors/contributors” could include:

Inputs and Activities	Direct products
Invested time and money	Number of new user accounts created
Program event(s)	Number of new editors trained
Participants

Activities and direct products of your program are important to track in order to evaluate what you did in order to assess the extent to which you did what you set out to do in terms of program implementation. However, measurable results are important for understanding your program outcomes and evaluating those results in terms of how they fit with your theory of change.

Measurable goals can vary a lot and may focus at different levels of impact based on the expected reach of your program impacts.

Program Objectives

Objectives provide guidelines for the accomplishment of goals:

Identifies the target group (Who and/or what will change?)
States the expected result (How will it change?)
Specifies the degree of change in measurable terms (How much it will change? How will the change be measured?)
Identifies the timeline for the change (When will the change have happened?)

Why Write Objectives

Objectives are the building blocks or steps towards achieving a program's goals. (Also can be aligned/mapped as chain of outcomes)
Objectives are specific and concise statements that state who will make what change, by how much, where and by when.

Examples of measurable result objectives (outcomes) toward the goal: “Recruiting new editors/contributors” could include results in participants skills and editing behaviors such as:

90% of program participants will demonstrate editing skills at the end of the program event.
70% of program participants will make at least one edit to Wikimedia projects within one month of their program participation.
Increase the number of new accounts created on [Wikimedia Project Name] who make their first edit by 5% by July 2014
30% of program participants who created new accounts will “survive” as editors three months after their program participation.
10% of program participants who created new accounts will be “active” editors for three months following their program participation.

How to Write Objectives

Three broad categories:

process (outputs/immediate outcomes)
outcome (short and intermediate outcomes)
impact (long-term outcomes)

The SMART Process to Developing Measurable Objectives

When writing goals and objectives, keep them SMART:

Using the SMART Process ~ S: Specific. Use specific rather than generalized language: Clearly state the issue, the target group, the time and place of the program.
Using the SMART Process ~ M: Measurable. Be clear in the objective about what will be changed and by how much. Setting this clearly at the start makes it easier to evaluate:
Using The SMART Process ~ A: Achievable. Be realistic about what the program can achieve in terms of the scale/scope of what is being done, the time and resources available:
Using the SMART Process ~ R: Relevant. Sometimes called “reasonable” or “realistic.” Objectives should reasonably relate to, and be relevant to, the goal and your program activities. Remember objectives represent the building blocks toward meeting goals:
Using the SMART Process ~ T: Timebound. Be clear in the objectives about the timeframe in which the program/activities, as well as expected changes, will take place:

Importantly, outcome objectives relate to various stages of outcomes and not processes.

Walk through an example [Note this is a hypothetical program]:

A program leader is planning a new program “Wiki Loves Notable Women” for improving coverage of notable women through a contest involving a month-long photo upload drive and series of online edit-a-thons to coordinate the creation of new, and improvement of existing, articles on notable women in terms of quality of coverage and images.

Process objectives would measure the activity of the program, for example:

To recruit new and existing users to participate in a contest to contribute photos of, and to articles on, notable women.

Applying SMART criteria, possible measures of Process/Activity Outcomes (a.k.a.”Outputs”):

X participants will contribute images and/or article content during the contest month.
X new contributors to Wikimedia.
Participation will include contributors to X or more wikimedia projects.
X notable women will be identified for article creation or improvement for the contest.

Outcome objectives measure the outputs and short-term, intermediate, and long-term outcomes.

Example outcome objectives include:

Short-term objectives which may be met immediately or soon after a program implementation and focus on changes in terms of direct products generated by the program participation and learning that takes place as part of a program (i.e., changes in awareness, knowledge, attitudes, skills, interest, motivations, or intentions).

Typically, outcome objectives are based on short-term outcome objectives, or the direct products that may be output as part of the program activity. An example of objectives for short-term outcomes related to the Wiki Loves Notable Women program might include:

To increase the availability of images of notable women on Wikimedia projects.

Applying SMART criteria, possible measures of immediate or “short-term” Outcomes:

X new images of notable women are added to the commons category during the month of the contest.
X articles created on identified notable women during the month of the contest.
X articles on notable women improved during the month of the contest.
X% of identified articles are created on at least one language project during the month of the contest.
X% of identified articles are improved during the month of the contest.
X new notable women articles on the projects/increased representation of women on wikimedia projects.
[If population total is known:] Increase representation of notable women by X%.
X% of identified articles created or improved are marked for translation within a month of the contest.

Sometimes you may also create specific objectives around intermediate-term outcomes which may be met within a few months following a program implementation and focus on changes in actions in terms of behaviors, practices, decision-making, policies, or social action.

Examples of intermediate outcome objectives include:

To increase the use of images of notable women on Wikimedia projects;
To increase the representation of women on wikimedia projects;
To increase the diversity of content on Wikimedia projects.

Applying SMART criteria, possible measures of intermediate outcomes:

X participants continue improving the content created as part of the contest after the contest ends.
X unique images of notable women added are used in Wikimedia projects within X months of the contest.
X images of notable women added are used across projects within X months of the contest.
X projects use the added images of notable women in their content within X months of the contest.
X articles created or improved as part of the contest are rated as Good Articles within X months of the contest.
X articles created or improved as part of the contest become Featured Articles within X months of the contest.
X new notable women articles are translated into X number of language projects within X months of the contest.

Long-term outcomes generally do not have specific objectives around them as they will not happen until a long time after a program ends; they are the impacts which take several months or even years to observe.

Long-term impact objectives focus on changes in the to the social, economic, civic, or environmental conditions originally targeted and evolve as objectives are achieved along the way.

Examples of Long-term Outcomes include:

To increase the accessibility and use of commons images of notable women in other media (e.g., other internet use, use in other publications);
To increase the diverse representativeness of women on Wikimedia projects;
Really (REALLY) long-term outcomes get right to your “mission”] To share in the sum of all knowledge.