Learning and Evaluation/Archive/Program Evaluation and Design/Budapest remaining questions

Budapest Remaining Questions Note answers here are preliminary and subject to change with review by the full PED team as well as with program leader feedback where perhaps more clarity in the question may be needed. Please feel free to share your comments, questions, and suggestions on the talk page.

Still have more questions? Ask on our Share Space Questions page where you can receive peer-to-peer support!

What is the difference between programs and projects?

Jaime's response: As we defined, a program is “a group of related projects and activities that share the same objective, are repeated on a regular basis, are based on a similar theory of change, and use similar processes and interventions to make that change happen.”

Key Characteristics of programs:

Shared objective: a group of related projects that share the same objective
Sustained: a group of related projects that are on-going or repeated on a regular basis
Model: a group of related projects that share a similar theory of change and use similar processes and interventions to make that change happen.

Here, projects are underlying tasks that make up your program.

Note: At the metalevel for Wikimedia in general, the term “project” also refers to the different Wikimedia projects (i.e., Wikipedia, Wiktionary, Wikiquote, Wikibooks, Wikisource, Wikispecies, Wikinews, Wikiversity, Wikivoyage, WikiCommons, Wikidata, and MediaWiki).

Sarah's response: A program is "a group of related projects and activities that share the same objective, are repeated on a regular basis, are based on a similar theory of change, and use similar processes and interventions to make that change happen." This means that a project is a part of a program. A program has three things that make it special: a shared objective (related projects that aim to achieve the same objective), sustainable (projects in the program are ongoing and happen regularly), and a model (there are other projects that might have similar theories of change and are similar in how they are executed).

For example:

In 2012, Wikimedia Sweden, Argentina, Poland and other chapters were running Wiki Loves Monuments projects that were all part of a international Wiki Loves Monuments program. These projects - all of the different WLM's produced by chapters - have shared objectives (to upload more photos of monuments). They are sustained because they all happen each September. They also have a model - they're based on the Wiki Takes/Wiki Loves model, which has participants photographing specific subject matter for upload to Commons. SarahStierch (talk) 19:27, 9 July 2013 (UTC)[reply]

Are goals different from outcomes? How do they fit into this structure?

Jaime's response: Considering the logic model presented, “outcomes” are synonymous with “goals”. The logic model tool, much like the SMART goals mapping process, is intended to aid program leaders in breaking down their general program goals into smaller, more tangible, and time-lined objectives. Where in the logic model the long term outcomes would be seen as more the goal(s) while short term and intermediate outcomes (as well as outputs) would often be considered the “objectives” underlying the goal(s).

Note that "outcomes" include generally include long-term goals/impacts, objectives and performance measures. See focus on outcomes diagram. {Here we have added examples from WMF mission and goals as they are readily accessible for exampling.

Often your short- and intermediate outcome targets are either objectives measured qualitatively or quantitatively OR performance measures which are quantitative measurements taken at different measurement times.

Importantly, objectives can also include programming "outputs" in addition to programming "outcomes." Importantly as the diagram depicts, both objectives and performance measures are how we measure our interim progress toward our stated goals through tangible and concrete measurable targets. Using a logic model to delineate these components more exclusively within your program scope and timeline helps to be sure not to leave gaps in your reasoning and/or story-telling ability. Conceptual frame adapted and modified from UW Extension (2003). Program Development and Evaluation Logic Model. Available here.

Sarah's response: I think Jaime's image is a really great tool in explaining the difference. A goal is basically a broader intention or aim you have, while outputs and outcomes are more specific and tangible aspects (objectives) underlying that goal. Importantly, outcomes are things that you can measure in order to monitor your progress toward the goal through tangible objectives or consistent performance measures.

For example, your goal with Wiki Loves Monuments is to increase coverage of monument photographs on Commons. Your outcome would be that participants uploaded 20,000 images of monuments in your country, which helps to meet your goal of increasing coverage of monument photos on Commons. SarahStierch (talk) 23:36, 10 July 2013 (UTC)[reply]

How far ahead is “Long term”/Separating outcomes?/ How do you define the scope/how will the evaluation redefine the scope?

Jaime's response: Long term outcomes refer to the lasting change(s) to the conditions that a program intends to change. They are the impact(s) that will be measureable if your program theory of change is successful. Depending on the scope of your intended impact (how much and what kind of change), your timeline for reaching Long term outcomes will vary. Generally:

Short Term Outcomes are the change you expect to occur either immediately or shortly after participants have completed the program.

Example: Learning (increased knowledge; increased understanding), Motivation (increased interest, increased intentions)

Intermediate Outcomes are the change you expect to occur after your short term outcomes are achieved. (Usually within three to twelve months following programming.)

Example: Action/Behavior (action happens from knowledge gained)

Long Term Outcomes are the changes you hope will occur over time given the intermediate outcomes have been evidenced. (Most often these are 3-5 year impact targets, however if the program is relatively short in duration and the vision is not far in scope these may be on a shorter timeline, say 1-3 years)

Example: Condition (conditions are changed as a result of those actions)

Although your program may specifically target, or you may only be interested in tracking, short term or intermediate outcomes, long term outcomes are generally those impacts that grant-makers often want to know about. They may even be beyond the timeline of your program and specific funding. Still, it is important to align your program theory of change all the way through from inputs, outputs, and short- intermediate, and long-term outcomes so that grant-makers may understand your programming needs and intended programming goals/outcome targets within a relatively consistent “big picture” frame. Often times, the owe-ness for assessing long-term outcomes/impact rests on the funder, however, a good theory of change map will guide grant-makers where to look .

How do we evaluate qualitative results?/How do we handle non-quantitative results?

Jaime's response: Depending on your qualitative data collection strategy, you may report qualitative results in various ways. Most often qualitative results include observations, interviews, facilitated group dialogues/focus groups, or a tangible product produced by a board or leadership group (strategic plan, meeting minutes, or other program documentation).

These different types of qualitiative data may be analyzed through multiple routes including descriptive statistics (count/number of persons, events, documents, etc.) as well as coded thematically to better understand the how and why of program action and outcomes rather than the what, when, who, and how much questions to which quantitative data collection and analysis are focused. Although many times qualitative analysis will provide quantitative results along with the qualitative content in reporting (i.e., 40% of all comments related to question X mentioned increased happiness or satisfaction, for example …. [listing of quotes])

Importantly, collection and analysis of qualitative data should still follow consistent criteria and should focus on your primary evaluation questions (answering whether you meet your intended outputs and outcomes). Note that another common application of qualitative results is their quantitative use. Follow-up quantitative collection of qualitative results is common as the information generated via qualitative inquiry regards the how and why of your different program achievements. What you learn about the potential how and why often becomes a quantitative question for future iterations of program participants to determine for how many, or how often, this is the case.

For some general tips and strategies for qualitative analysis click here.

UserMetrics API: When will it come and how workable will it be?

Jaime's response: The replatformed UserMetrics API (UMAPI) will become available for use at the end of July. The tool’s name will now be Wikimetrics. The analytics team is developing the new version to have a user-friendly interface that will be adjusted to meet the needs of programs in the most simple and direct way by allowing you to upload or choose your cohort and allowing you to select a metric and then prompting you to select your date start and end parameters. Other available parameters for the metric should also appear for you to see so that none are operating behind the scenes as unknown defaults.

Sarah's response: It's been renamed Wikimetrics, which I like better, actually. On that note, it looks like it might be ready for community members to play with it no later than the beginning of August, fingers crossed. SarahStierch (talk) 17:19, 12 July 2013 (UTC)[reply]

Who will execute the programs in the long term?

Jaime's response: Question seems unclear. Program leaders will continue to execute programs. This includes chapters, individuals, and other programming groups who wish to do so in their local Wikimedia communities.

How to drive/execute several projects at the same time (how to order and prioritize)?

Jaime's response: In order to facilitate your prioritization in terms of program tracking as well as evaluation data collection and analysis we plan to continue to work with program leaders to identify important commonalities as well as important uniqueness’s that you may want to focus upon in telling the story of your individual programs. Importantly, building capacity for your programming and program evaluation will need to be a consideration in your budgeting and staffing. This additional effort will indeed require capacity enhancement both in skills and in staff funding and should be reflected in your grant proposals. We anticipate that further guidance on this matter will develop as our overall conversation about evaluation evolves. You likely will not be able to measure everything in your theory of change/logic model, what you do focus on may also shift as you meet or don’t meet certain outputs and/or outcomes in the implementation of your program. Just as different types of evaluation may be required at different times, you may need to focus in on different pieces of your theory of change as you seek success in programming.

How are we going to document everything?

Jaime's response: You simply are not going to be able to document everything. We will all work together in the coming months to try and prioritize the inputs, outputs, and outcomes that will be most important for reporting the story of your programs. You may also choose to maintain an eye on some of the particular inputs, outputs, and outcomes you find to be uniquely important aspects to your program and context.

Sarah's response: It will be tough to document everything. I do my best to keep notes on how the process is going with everything I do program wise - even though it is hard. Sometimes I keep a wiki page devoted to the process - my timeline, my goals, intended impact, and lessons learned. Also - take photos, record programs and activities within them, keep your own diary - things like that can totally help when it comes down to evaluating how a program went. Even keeping a blog, Twitter, or Tumblr can be a good idea, and don't forget those disgruntled Facebook posts you might make about how an event is coming along :) SarahStierch (talk) 17:21, 12 July 2013 (UTC)[reply]

How often are we going to have these workshops?

Jaime's response: Face-to-Face workshops will always be less frequent than online meet-ups, and it has not been determined how we will proceed with face-to-face meetings with this group versus bringing along another group and developing more online group meet-up strategies for the in-between. It may be that the most manageable time for us to regroup in-person will happen more naturally at the two existing annual meetings. However, we will work to be as responsive to the groups needs as we can be within our budget. Our meeting practices will aim to keep everyone moving in a coordinated direction.

Sarah's response: I'd love to see more workshops, for sure! But, yes, it will be tough to have a lot of workshops due to budget and accessibility. If you have an event taking place in your region, or your chapter is doing an event, or if you have interest in doing a workshop or having us lend a hand, do let me know. I'm happy to see how the PE&D team can support you with materials, or participate ourselves. And yes, we will be doing monthly online events. More soon on that :) SarahStierch (talk) 17:26, 12 July 2013 (UTC)[reply]

Are we going to follow-up as a group?

Jaime's response: We will stay together as a group for our scheduled meet-ups online and off, yes. See above also.

Why did we choose this evaluation model?

Jaime's response: Although we have suggested to emphasize a participatory-empowerment and utilization-focused blended approach to the overall design of Wikimedia programs’ evaluations in order to be consistent with the values and mission of Wikimedia, the specific evaluation model you choose will always be up to you.

Importantly, the “Logic Model” and “Theory of Change” are neither an evaluation model nor an evaluation plan. Rather, the logic model and theory of change are a communications model for mapping out the how, what, and when of your program inputs, outputs, and outcomes so that we can share a common definition and understanding of each of the common elements that you may, or may not, choose to include in your individual evaluation model(s) and plan(s).

Is the model intended to integrate with project management, will it help with decision-making?

Jaime's response: The logic model is a detailed outline of various elements you may wish to monitor and use in project management and decision-making. The logic model categories are broken down into the specific components that should be transferable to nearly any project management or localized evaluation model so that while you customize your program planning in terms of project management and local evaluation interests, we may retain a common and consistent categorical understanding of the basic elements across programs and programming contexts for our shared understanding.

What will be the next step?/How am I going to apply this?/How do I apply the theory of change?/How can I translate this to the community?

Jaime's response: Once you have a chance to review the draft logic models on our shared space we will work to prioritize the inputs, outputs, and outcomes within each program and the relevant evaluation questions key to telling a common story. In order to stay customized to your own programming context you will likely want to take these prioritized elements, along with your unique interests, and develop your own localized logic model and evaluation questions as needed and from there, we will work to develop additional tools for tracking and inquiry from participants. This will be a little different for everyone and so we will be available to help you as you need support in the process.

How can I teach others (budget?)/Would the WMF pay for educational events?

Jaime's response: Building capacity for program evaluation will need to be a consideration in your budgeting. This additional effort will indeed require capacity enhancement both in skills and in staff funding and should be reflected in your grant proposals. We anticipate that further guidance on this matter will develop as our overall conversation about evaluation evolves. Still, not all stakeholders need to have a full understanding of the underlying practices in mapping out your programs. Often presenting the completed logic models and the clear goals and objectives contained within it are sufficient for larger community dialogue, while anyone tasked with mapping and evaluating would be best served by a presentation similar to what was done at our workshop. The presentation slides will be available soon, but more importantly, we are working to push the presentation to an online training module in the wiki format soon.

Sarah's response: Like Jaime said, over time you'll find tons of great resources here - from not just the PE&D team, but from the Learning & Evaluation team as well, so you can make sure not only your programs (like edit-a-thons and workshops) are making an impact, but, that you as an individual, and your chapter (if you are involved in one) are able to successfully host events and activities related to PE&D. If you are planning anything, you can always ask me and I'll do my best to make sure that the PE&D team fufills your needs to make the most of any workshop you chose to host.

Also, if you need funding for an event or workshop you can apply for funding through the Wikimedia Foundation Grants Program. There, you can get money to support workshop materials, getting a venue, hiring a specialist, or whatever you need to make the most of your event. And of course, if you do a workshop, don't forget to evaluate it ;) SarahStierch (talk) 17:31, 12 July 2013 (UTC)[reply]

How important is the language transfer to my chapter?

Jaime's response: Sharing a common language around evaluation will be key to dialogue between program leaders, evaluators, the Program Evaluation and Design Team, and the Grantsmaking department. Beyond that, shared terminology is always helpful, but may not be as necessary to your larger chapter and community groups as there should be a person available (i.e., you and possibly your evaluation teammates) to decipher and translate when confusion may occur. Importantly, in order for us to be responsive to your needs for resource development and support it is important that we understand if you are asking bout tracking an input or output, vs. measuring an outcome and such. It will also be important for grantsmaking to understand your scope in terms of what level of outputs and outcomes you are seeking for the inputs you put in to your programs.

Sarah's response: We're hoping that community members will help translate these materials, including the glossary. You can find the glossary, which has been tagged for translation, here. It's important to have a shared vocabulary and to have resources translated into your language. Therefore, when we do communicate here on wiki or in person we can all be on the same page - meaning we all can know what one another is talking about when they say "outputs" or "Activities" no matter what the language is :) SarahStierch (talk) 17:33, 12 July 2013 (UTC)[reply]

Could this have been explained in simpler terms?

Jaime's response: The more constructive feedback we receive and the more voices participating in discussing around the confusion points will help s to continue to unpack the presentation content into more and more easily digestible units. We appreciate everyone’s efforts as we seek to build a new and highly complex shared vocabulary around program evaluation.

Sarah's response: Totally! And we're working on that. It's still a challenge for me, and I'm new to program evaluation and a native English speaker. One of my roles, as community coordinator, is to do my best to make sure the language and materials are easy to understand for everyone around the world. So yes, we're working on, as Jaime said, building a shared vocabulary, but also making it as accessible to everyone around the world - even those not involved in Wikimedia. SarahStierch (talk) 17:34, 12 July 2013 (UTC)[reply]

Will it lead to my fears or hopes?

Jaime's response: Asking questions about a program’s successes and challenges always leads to discovery of both. In order to lead to hopes of program sustainability and growth it will be important to stay true to your goals and adjust your programming as needed. Through iteration after iteration of your program it is expected that successful evaluation will lead to successful adjustments in programming and implementation that will lead to continuous improvement and increased attainment of those goals. If you do not mind the findings of your evaluation strategies and do not adjust your programming as needed, evaluation very likely can demonstrate a lack of achievement that does not improve over time, evidence of a faulty theory of change or a program targeting inappropriate outcomes more likely. Not many funders will continue to fund unproductive programs ad infinitum so, fears could be met if year after year you do not adjust either your program strategy or expected outcomes to demonstrate success. That is not the intention of the Program Evaluation and Design Team initiative. We are here to help you collect information to guide programs toward success. We are here to help you tell the story of your program, successes and challenges, so that we can learn and grow from one another’s experiences. This evaluation approach is focused on “Growing the Awesome” and we hope that while innovative solutions to challenges do come, we do not get hung up on fixing problems but expanding what works well innovating high impact programs.

How to include the movement in this discussion? Which concepts?

Jaime's response: This is a question that really needs a group discussion. Most often logic models are shared with the larger group of stakeholders once program leaders come together on the overall elements. It may be important to learn from community stakeholders what their perception of desired outcomes and their level of buy-in for the various targeted outcomes is before prioritizing, or after, allowing for potential additions and/or revisions that may be generated from the dialogue. Each of you has a different community and they may all have different preferences for inclusion in the planning, decision-making, and/or review process regarding specific outcome goals. It will be good to feel out your particular community of programming associates, participants and community members to see what may be most appreciated. Importantly, some of the more qualitative outcomes are often those understood by program participants and other stakeholders with a different vantage point of the program and so it is likely important that you do engage that broader audience if you want to be sure to get the biggest picture represented before narrowing in on prioritized indicators and particular evaluation questions that you will actually pursue. [I propose that this question become a larger discussion on our shared space as it was also one of our Pickle jar items]

Will decision-making be top-down?

Jaime's response: Every program leader is in charge of their own evaluation and decision-making. The Program Evaluation and Design Team has the intention of being as democratic and representative as possible within the task of building capacity and tools across the variety of Wikimedia programs so that all program leaders can achieve that. Unfortunately, even through democratic process, not all needs can be met, certainly not within a desirable time-frame especially for some of the more unique program interests and needs. Importantly, we hope that growing a shared understanding and way of communicating about the inputs, outputs, and outcomes components within each programs theory of change will allow you to clearly communicate the story of your program as it both fits your expectations and alternatively unfolds in it’s own unique way while coming to some agreements about where programs cross paths or come together in terms of strategies and measures. Whether you discover you made the impacts you intended or that you made an impact that you had not imagined, it will be up to you as to how you use the data to drive your next programming steps. Where we find alignment across programs is where we can eventually compare across program implementations and contexts; where there is sameness and/or difference will be a guide to identifying promising practices that may be helpful to others as your programs work to build toward the overall mission of Wikimedia. The Program Evaluation and Design Team will do our best to facilitate the overview for such an exchange of ideas and practices, but programming decisions, and even your particular decisions regarding specific evaluation questions and models, will rest on you all as program leaders. Our goal is to help you build capacity to collect and analyze measurable outcomes so that you can make data-driven decisions and demonstrate strategic steps toward growing your programming success. Your evaluation should help you to do this specifically by guiding you toward what works and why, while at the same time helping you to discover what doesn't work. You as program leaders will be responsible to make your own program expansion/modifying/reduction planning decisions based on the outcomes your programs do or do not demonstrate and the shared knowledge we are able to develop and extend across the community.

Sarah's response: Just a quick comment - no, it won't be. Wikimedia Foundation isn't forcing anyone or any organizations to do this. If you don't do program evaluation, it doesn't mean you will never ever recieve funding, or that you're doing a bad job as a program leader or community member. When Frank and I talked about forming this team, we both had a strong desire to see how we - as community members - can do programs we like doing (Frank likes to do contests, I do GLAM stuff and edit-a-thons) better, and not "waste time" putting time and energy into something that isn't making an impact (but if I have epic fun doing it, I'll still keep doing it ;) - and evaluating fun is part of the evaluation process!). We hope that with PE&D, we can help you make better decisions on what you put your time and money into (or movement money) and to make sure the work you do is celebrated and improved upon the world over. And again - WMF is not making anyone do this. Promised. SarahStierch (talk) 17:40, 12 July 2013 (UTC)[reply]