Grants:APG/Proposals/2014-2015 round2/Wikimedia Armenia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal. Members of the FDC review each proposal in its entirety, including relevant background documents, inputs from WMF grants officers, the learning and evaluation team, and finance staff. They also analyze the detailed discussions surrounding each proposal that include questions and comments from community members. FDC staff do not participate in the actual decision-making process of the FDC; their primary role is to identify key strengths and concerns for the FDC to consider while making funding recommendations.

This staff proposal assessment, which was refined after the first two years of the FDC process, consists of three parts: (1) Overview, including a financial summary and a summary of strengths and concerns; (2) Staff proposal assessment narrative, including a nuanced analysis of the programs, context, and the feasibility of each plan; (3) Staff proposal assessment rubric, assessing each proposal across three dimensions by identifying areas of strength and concern. Read more about the methodology behind the Staff proposal assessment rubric on this page.

Staff proposal assessments do not represent the views on any one staff person, since all FDC staff (unless one of them recuses for some reason) work together to complete each staff proposal assessment, and may consult with other key WMF staff and other experts as needed. While assessments will not be changed after they are published except to correct significant factual errors, organizations are welcome to share responses on the discussion page of each staff proposal assessment if they would like to provide the FDC with additional information or perspectives on the assessments.

Overview for all assessments

As in Round 1 of 2014-2015, proposals in 2014-2015 Round 2 are being assessed in the context of some preliminary impact data for grants and programs over the last two years, and our current and future approach is being guided by this analysis. In particular, we have been rigorous with our assessments of the Wikimedia organizations with the largest budgets and requests, as we expect them to be delivering outcomes to scale.

For example, we were able to access data from the first round of impact reports for grantees entering the APG process in 2012-2013 Round 2 (i.e. Wikimedia Norge and Wikimédia France), as well as progress reports received from the current grants (i.e. Q1 and Q2 reports) and project grant reports for all organizations but Wikimedia Italia, which is new to the APG program and did not receive significant project funding in the past. This allowed for a more detailed assessment of proposed goals against progress in the current year, as well as the current year’s progress against the current year’s goals. In some cases, we were also able to compare the current year’s goals and performance with past achievements. A move toward the inclusion of more easily comparable metrics in reports also assisted with this analysis.

There are a few overarching themes we noticed in this year’s proposals that we want to highlight. Please note that any examples used are meant to be illustrative, not exhaustive.

Train the Trainer approaches

In this round, we noticed a focus on Train the Trainer (TTT) approaches, which builds on the interest from Round 1. With a TTT approach, organizations target and train a group of participants on specific topic areas so they may in turn train others. If designed and implemented well, this program approach may have potential to expand reach in the movement and improve relevant skills beyond an organization’s direct sphere of influence.

This approach, which takes significant resources and organizational focus, requires more clearly articulated strategies, documented/centralized resources, and measurement and evaluation of the results. Most organizations are not yet evaluating skills developed, results of trainings after the original trainings, or attitudinal shifts, and so it is difficult to understand the impact of these programs. Furthermore, we are concerned not to see plans for sustaining engagement with these trainees after trainings.

We encourage organizations to document results and resources invested in these programs. We also encourage organizations to develop engagement plans before new training programs are implemented. Finally, we also encourage organizations start small before making the decision to invest in programs more significantly.

Metrics and evaluation

As in Round 1, we realize that all Wikimedia organizations in this round are improving their measures of success and attempting to demonstrate impact. This work is not easy; we recognize the effort. For example, we are pleased to see some (including Wikimedia Armenia, Wikimedia Norge, and CIS) offer baselines and projected outputs and outcomes. This is a significant improvement from 2013-2014 Round 2 proposals.

However, not all programs include metrics, particularly around outcomes. We are very concerned, particularly in the case of the larger organizations, about the lack of a baseline and project outcomes beyond outputs. Quantitative and qualitative evaluation methods are both important. The Wikimedia Foundation will provide continued support to organizations who request it to develop capacity in evaluation. On the quantitative side, we encourage organizations to establish systems to track the global metrics, as they will need to report on in order to understand their work in a broader context, and also to continue to carefully consider what metrics will be most relevant in their contexts.

Diversification of funds and resources

Especially for organizations with larger budgets, we are monitoring reliance on APG funding. Organizational reliance on APG funds (expressed as a total percentage of revenue) may be increasing for some, and in other cases may not be decreasing at a sufficient pace. In some cases, large organizations are not meeting fundraising targets or cultivating sustainable longer term institutional partnerships. We are particularly impressed with organizations in the global south who have been able to cultivate donors.

All organizations in this round are proposing significant total budget growth. Each organization’s growth must be examined in context, but in some cases this rapid growth does raise capacity concerns.

As we noted in Round 1, the diversification of resources is not only needed in order to maintain the sustainability of each individual organization and the global movement, it is also a way to build awareness of the Wikimedia mission.

Lobbying and political advocacy

In Round 1, we saw organizations express significant interest in policy, advocacy, and awareness.

In Round 2, Wikimedia Italia has proposed to engage a lobbyist to address legal threats to its program. Any lobbying work requires some oversight from the Wikimedia Foundation. We will monitor this work over time.

Increase in program activities and organizational capacity

In many cases, organizations are planning a large number of programs, with a many activities. This raises concerns around organizational capacity. Beyond the activity and staff expenses for each program, programs represent a significant investment of organizational focus. We encourage organizations to group programs or prioritize them appropriately in order to present a clear picture of what they are trying to achieve and why, and in order to reduce strain on volunteer boards, staff, online contributors, and offline volunteers active in program areas that can result from too many dispersed activities. We encourage organizations to focus on their strengths rather than to experiment in every area. In this way, organizations can experiment with thoughtful adaptations of existing approaches based on local context and the organization’s expertise, and with small scale pilots with potential to scale up.

As we noted in Round 1 around proactive and responsive community engagement strategies, we realize it is not easy to achieve a good balance of responding to community needs and leading strategic work for the movement. Refining focus will be an ongoing process.

Diversity and work on projects other than Wikipedia and Commons

As in Round 1, an interesting development is the increasing focus on Wikimedia projects other than Wikipedia. For example, we have seen CIS and Wikimedia Armenia integrate work on Wikisource and Wiktionary into key programs.

As in Round 1, we are also assessing proposals across the dimension of diversity, and we have seen Round 2 organizations focusing on this in their contexts through both a regional and a gender-focused lens. Some organizations have diversity at the core of their missions (e.g. CIS, Wikimedia Armenia), some are approaching this work through a cross-cutting approach (e.g. Wikimedia Italia) and others have designed specific programs to address diversity (e.g. Wikimedia Norge).

Overview

Overall assessment and eligibility

Organization	Wikimedia Armenia
Eligibility	Confirmed

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$90,952	$123,895	36%
Budget	$105,952	$138,895	31%
Staff	0	1	NA

Overview of strengths and concerns

This section summarizes the key strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

Wikimedia Armenia is effectively engaging new and existing online and offline volunteers through its programs, which provide pipelines for volunteer development
Proven ability to budget for and plan programs effectively, demonstrated through its Project and Events Grants, which is especially impressive considering the all-volunteer composition of the organization.
Thoughtful adaptations of existing programs and experiments with new ideas that appear to be well-suited to Wikimedia Armenia’s context

Concerns

This proposal includes a significant number programs, and the organization’s focus may be diffuse, which may be a challenge given the smaller size of the organization
Managing staff for the first time may be a challenge for Wikimedia Armenia’s core volunteers.

Staff proposal assessment narrative

This section takes an in-depth look at the strategy behind this proposal, this organization's broader context, and the feasibility and risks of this plan.

Context and potential for impact

This section takes a close look at this organization's context, since there will be unique factors specific to this organization or to its environment that enable this plan to have impact or that make this plan less likely to have impact. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact? Are there risks and opportunities presented by this organization's environment? Are there extraordinary events or other factors that need to be considered as part of this organization's context?

Wikimedia Armenia has developed programs and approaches that are particularly adapted to its context, and it has tailored these programs and approaches to achieve more impact by leveraging its volunteer community.
Wikimedia Armenia has strategically cultivated an engaged community of offline volunteers and new online contributors, who have maintained an impressive level of engagement with Wikimedia Armenia’s activities over time.
Furthermore, Armenia is well placed to address global content gaps through its work on Wikisource and digitization.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed? Does it raise any concerns?

Wikimedia Armenia has a strong track record with respect to its past project and event grants. It has proven that its programs achieve results in its local context and that it has the ability to execute on its programs and manage program finances effectively.
In addition, the organization has successfully raised $30,000 from the Fundação Calouste Gulbenkian, which is a notable amount of revenue for an organization of Wikimedia Armenia’s size and geographic context.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed? Does it raise any concerns?

Wikimedia Armenia has shown a strong ability to manage successful programs that lead to results, with an impressive ability to retain editors over time.
For an organization of its size, it has shown a good capability in financial and impact reporting, and is a stable and well-run organization with an active core of volunteers.
Wikimedia Armenia has not had experience managing staff, and this will be a new area for them that should be monitored closely.
Wikimedia Armenia is a learning organization that adapts its programs in order to apply lessons learned, both from failures and successes. An example of this can be seen in the evolution of its successful WikiCamps programs and in the relationships between its programs, which help to establish a pipeline to grow new volunteers and contributors (e.g. WikiClubs laying the groundwork for education programs in new communities).

Strategy, program design, and program implementation

This section takes a close look at this organization's plans for its programs, including its past performance with program implementation. In this section, we may also identify specific programs that seem to have particularly high or low potential for impact. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? Is this organization proposing significant changes to its strategy, or are the strategic plan and annual plan consistent with past strategies? Is this organization's strategy and are this organization's programs aligned with the Strategic Priorities of Increasing participation, Improving quality, or Increasing reach?

For an all-volunteer organization of its size and stage of development, Wikimedia Armenia has shown an adequate level of strategic planning. In the future, it may consider developing its programs in line with a longer term strategy.
The programs proposed in the current plan are consistent with Wikimedia Armenia’s past programs and approaches. This plan does not represent a significant change in overall approach, although the number of activities is increasing. Wikimedia Armenia is continuing its focus on programs that target and benefit youth, as it has assessed that this approach is effective in its context.
Wikimedia Armenia’s programs are closely aligned with increasing participation, with a focus on programs that bring in and retain new contributors. Digitization work is well-aligned with content, and Wikimedia Armenia is doing programs that broaden its geographic reach by laying the groundwork for future communities of contributors and offline volunteers (e.g. WikiClubs).

Program design

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked?

In this proposal, Wikimedia Armenia has produced SMART objectives for its programs and has clear plans to measure results. Baselines are included, to put proposed targets in context.
Wikimedia Armenia has incorporated the advice of the Grants Advisory Committee and PEG program staff in developing its past work and activities to drive toward greater impact, and has also incorporated advice from the Program Evaluation team at the Wikimedia Foundation in developing this proposal.
Throughout the proposal, Wikimedia Armenia articulates strong links between offline programs and online impact.
Despite the strength of its programs, Wikimedia Armenia’s ambitious plan with a great number of programs may make it difficult for the organization to remain focused.

Specific programs

Based on proposed plans and past work, which programs may have the highest potential for impact corresponding to the amount of funding requested, and which programs may not have impact corresponding to the funds requested?

WikiCamp is Wikimedia Armenia’s flagship program, and its programs with youth have been particularly effective to date. WikiCamp also includes the most robust set of relevant targets, which may be appropriate as it is Wikimedia Armenia’s most expensive activity by far. Wikimedia Armenia cites an impressive 55% of campers still active as Wikipedians one year after the the camps.
International Involvement is Wikimedia Armenia’s next most expensive program and presents the least relevant targets, and the fewest logical links to online impact.
While Wikimedia Armenia is cultivating a partnership with an external donor on its Western Armenia work, we will follow this work closely to make sure it is in line with movement goals. Work on Western Armenian (including a dispersed diaspora community) is experimental, and we are looking forward to learning from the results.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Wikimedia Armenia is a recently recognized chapter and a first-time applicant to the APG process, and so significant growth in budget and APG funding is reasonable in this context. It is important to note that the grant request is still a significant amount in the Armenian context.
Wikimedia Armenia has provided a detailed budget and clear financial information, and has demonstrated capacity to budget effectively through its performance on past project grants.
This budget is focused on impact. Wikimedia Armenia has an exceptionally low proportion of staff expenses to overall expenses, and expenses are weighted toward impactful programs.

Feasibility and risks

How likely is it that this organization can implement this plan successfully if funded? Is this plan likely to achieve impact corresponding to the funds requested?

It seems likely that Wikimedia Armenia can implement this plan successfully if funded, and that impact would correspond to the funding requested.
A clear and proven strategy around volunteer engagement, well-articulated logical links between offline work and online impact, SMART program design, strong strategic alignment to increasing participation, and stable and effective volunteer leadership all enable this plan to succeed.
Wikimedia Armenia has received funding from at least one institutional partner, which is significant in its context.
Wikimedia Armenia is expanding its budget significantly and is hiring staff for this first time. Wikimedia Armenia’s ability to successfully manage staff is not yet proven, and this will be a significant burden on its hard-working volunteer board. At the same time, the organization has proposed a sensible and gradual approach to increasing its staff by hiring two part-time staff people to focus on programs and administration, respectively.
As program activity is expanding at the same time that oversight work will increase, it may be a challenge for Wikimedia Armenia manage this ambitious plan.

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design and strategic alignment, (2) Organizational capacity and effectiveness, and (3) Budgeting by identifying strengths and concerns in each of these areas.

In each assessment, an organization’s context is taken into account and so criteria may be applied differently in different contexts. For example, a smaller organization with less experience will not be expected to have a formal learning and evaluation program that a larger more experienced organization might have, and so what may constitute a major strength for a smaller organization in the same category may not constitute a major strength for a larger organization. With this approach, the criteria listed here are not weighted equally, and so an “overall assessment” cannot be determined simply by adding up the numbers of strengths and concerns identified for each dimension. Also note that the scale here is not linear and so we do not assign any numerical values to the assessments given for each criterion. More information about each criterion is shared in the assessment. We believe a framework like this is helpful, as it ensures that each proposal is assessed consistently, even as they are applied with contextual nuance.

The dimension “Program design and strategic alignment” addresses many high-level elements, focusing on how likely programs are to achieve progress toward some of the Wikimedia movement's strategic priorities. Programs need to show clear links between strategic priorities and program objectives, program objectives and targets, and targets and program activities. In addition, programs should focus on areas that will impact participation, reach, and quality on the Wikimedia projects, with an emphasis on approaches and programs that increase diversity.

Meanwhile, the dimension “Organizational capacity and effectiveness” looks at the ability of an organization to achieve its plan based on both past experience (as an indicator of future success) and current practice. An effective organization both learns from past experience and contributes to movement learning in the process. This dimension also includes effective and stable leadership that models good practices, and a rational approach to growth that is supported by the organization’s strategy and context.

Finally, we analyze the organization’s past experience with and current plan for budgeting. Here, we look at how the organization has performed against its plans and budgets in the past.

To complete the assessment, we assess each of the criteria in each category according to the following scale, which is not necessarily linear:

Major strength
Strength
Neither a strength nor a concern
Concern
Major concern

Criterion	Assessment	Description
Program design and strategic alignment
P1. Strategy and alignment	Strength	Programs objectives are strongly aligned with the Wikimedia strategic priorities of increasing reach, improving quality, and increasing participation, with a level of strategic planning appropriate for an organization of this size. Wikimedia Armenia can improve in future years by developing and articulating a longer term strategic plan to guide its work.
P2. Targets and logic models	Major strength	All programs have specific, measurable, time-bound objectives with clear targets, and programs present clear links between planned activities and desired results. Objectives are relevant and present clear links to online impact.
P3. Needs assessments	Neither	Program objectives are based on needs assessed in consultation with stakeholders, but needs and the process by which they are assessed are not yet clearly articulated in the plan.
P4. Potential for impact at scale	Strength	Given the comparatively small size of the funding request, programs could lead to significant impact on the Wikimedia strategic priorities of increasing reach and increasing participation at scale. Particularly, Wikimedia Armenia is engaged in successful experiments around pipelines for developing a vibrant community of offline and online volunteers, and in growing new communities of Wikimedians in different regions (e.g. Diaspora communities).
P5. Evaluation methods	Strength	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Wikimedia Armenia has proven ability to do this. Plans for evaluation and measuring can be more clearly articulated for each program in the future, as Wikimedia Armenia grows its capacity in this area.
P6. New ideas and approaches, or thoughtful adaptations	Major strength	Wikimedia Armenia is engaged in thoughtful experiments and approaches, and continues to adapt and learn from this work. It particularly excels at developing approaches unique to its context (e.g. Wikiclubs and Wikicamps).
P7. Diversity	Strength	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served. Particularly, Wikimedia Armenia's leadership, volunteer base, and base of online contributors already includes significant female participation and Wikimedia Armenia is well-placed to contribute to global content gaps.
Organizational capacity and effectiveness
O1. Past results	Major strength	Through its grants in the PEG program, Wikimedia Armenia has proven success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Stability and leadership	Major strength	Stable, effective leadership and good governance will enable this plan to succeed. Wikimedia Armenia has been a stable and effective all-volunteer organization in the past, and is requesting to transition to an organization with staff in the coming year.
O3. Learning	Major strength	Wikimedia Armenia is addressing risks and challenges effectively, is learning from and documenting past experiences, and is applying learning thoughtful to improve its current and planned programs. Wikimedia Armenia has shown a good understanding of its context and its audience, and has leveraged this knowledge to plan effective programs.
O4. Improving movement practices	Strength	Wikimedia Armenia is growing in its ability to effectively share learning about its work with the broader movement and beyond, at an appropriate level in its stage of development, including collaborations with other chapters in the CEE region and nearby.
O5. Community engagement	Strength	Wikimedia Armenia effectively engages key communities and volunteers in the planning and implementation of its work. Wikimedia Armenia's work is driven by a dedicated and engaged volunteer base, that is consistently cultivated by the organization. Wikimedia Armenia can show greater evidence of engagement with online contributors as it works in the future to show clear links between online impact and offline work.
O6. Capacity	Strength	While Wikimedia Armenia has engaged a stable volunteer base to enable this plan to succeed, their ability to effectively manage staff is not yet proven. Wikimedia Armenia has also demonstrated ability to fundraise externally.
Budget
1. Past budgeting and spending	Major strength	Wikimedia Armenia has a history of budgeting realistically and managing funds effectively in the past, especially for an all-volunteer organization of its size.
B2. Proposed budget is realistic	Strength	Wikimedia Armenia's proposal includes a budget that is comprehensive and clear, and that corresponds to the activities proposed, especially for an all-volunteer organization of its size.
B3. Budget is focused on programmatic impact	Major strength	Wikimedia Armenia has a history of spending funds effectively and has presented a budget that is significantly weighted toward impactful programs.

This staff proposal assessment is the work of FDC staff and is submitted by: KLove (WMF) (talk) 22:03, 8 May 2015 (UTC)

Staff proposal assessment rubric description

This rubric assesses annual plan grant proposals across the three dimensions of (1) Program design and strategic alignment, (2) Organizational capacity and effectiveness, and (3) Budgeting by identifying strengths and concerns in each of these areas.

In each assessment, an organization’s context is taken into account and so criteria may be applied differently in different contexts. For example, a smaller organization with less experience will not be expected to have a formal learning and evaluation program that a larger more experienced organization might have, and so what may constitute a major strength for a smaller organization in the same category may not constitute a major strength for a larger organization. With this approach, the criteria listed here are not weighted equally, and so an “overall assessment” cannot be determined simply by adding up the numbers of strengths and concerns identified for each dimension. Also note that the scale here is not linear and so we will not assign any numerical values to the assessments given for each criterion. More information about each criterion is shared in the assessment. We still believe a rubric like this is helpful, as it ensures that staff analysis of each proposal is based on the same criteria, even as they are applied in a nuanced way.

The dimension “Program design and strategic alignment” addresses many high-level elements, focusing on how likely programs are to achieve progress toward some of the Wikimedia movement's strategic priorities. Programs need to show clear links between strategic priorities and program objectives, program objectives and targets, and targets and program activities. In addition, programs should focus on areas that will impact participation, reach, and quality on the Wikimedia projects, with an emphasis on approaches and programs that increase diversity.

Meanwhile, the dimension “Organizational capacity and effectiveness” looks at the ability of an organization to achieve its plan based on both past experience (as an indicator of future success) and current practice. An effective organization both learns from past experience and contributes to movement learning in the process. This dimension also includes effective and stable leadership that models good practices, and a rational approach to growth that is supported by the organization’s strategy and context.

Finally, we analyze the organization’s past experience with and current plan for budgeting. Here, we look at how the organization has performed against its plans and budgets in the past.

To complete the assessment, we assess each of the criterion in each category according to the following scale, which is not necessarily linear:

Major strength
Strength
Neither a strength nor a concern
Concern
Major concern

Criterion	Description
Program design and strategic alignment
P1. Strategy and alignment	Programs objectives are strongly aligned with the Wikimedia strategic priorities of increasing reach, improving quality, or increasing participation. The organization has a strategic plan in place.
P2. Targets and logic models	Programs have specific, measurable, time-bound objectives with clear targets, and programs present clear links between planned activities and desired results.
P3. Needs assessments	Program objectives are based on needs assessed in consultation with stakeholders, including the communities and volunteers that are working together with this organization to achieve its program objectives.
P4. Potential for impact at scale	Programs could lead to significant impact on the Wikimedia strategic priorities of increasing reach, improving quality, or increasing participation, at scale and corresponding to the amount of funds requested.
P5. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems.
P6. New ideas and approaches, or thoughtful adaptations	Programs will test new ideas and approaches or will thoughtfully adapt ideas and approaches that may work in this organization’s context.
P7. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational capacity and effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Stability and leadership	Stable, effective leadership and good governance will enable this plan to succeed. Any leadership transitions are managed effectively.
O3. Learning from the past	This organization is addressing risks and challenges effectively, is learning from and documenting past experiences, and is applying learning to improve its current and planned programs.
O4. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond.
O5. Community engagement	This organization effectively engages key communities and volunteers in the planning and implementation of its work.
O6. Capacity	This organization has the resources and ability (for example, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Proposed budget is realistic	This proposal includes a budget that is comprehensive and clear, and that corresponds to the activities proposed.
B3. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.