Grants:APG/Proposals/2017-2018 round 2/Wikimedia Armenia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$302,415	$267,454	-11.56%
Budget	$319,874	$342,668	7.13%
Staff (FTE)	4.75	5.50	15.78%

Overview

This section summarizes the themes that emerged from the proposal, which are explained in the detailed narrative section of this assessment.

Themes

• WMAM is a resilient organization that is able to handle transitions in leadership well, and they are strengthening their foundation under a new ED.
• This proposal includes better program design and well-explained context and rationale.
• Evaluation at the activity level is not always relevant and will be difficult to implement.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

WMAM continues to evolve in a very positive environment for Wikipedia and the Wikimedia projects.

WMAM continues to take advantage of opportunities for partnership and funding in Armenia and beyond, as well as opportunities to work with the broader diaspora community outside of Armenia.

Many of WMAM’s programs are very well-adapted to their particular cultural context, and WMAM takes time to examine trends in their broader social context that may present opportunities or challenges in their work. The continuation of Wiki Clubs and the way they are anchored in everyday life in rural Armenia shows how close WMAM is to their community and the general public, whom they teach how to become part of the wiki communities. Their continuous work to adapt wiki clubs also shows how they are listening to their environment.

While the Armenian language Wikipedias and communities already have a good gender balance, we encourage WMAM to think about how to define diversity in a way that is relevant in their own context.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

While WMAM has historically achieved impressive results in terms of involving new and experienced contributors and producing large amounts of content to grow the Armenian language projects, they have also recognized that they need to deepen their impact and their approach to evaluation in some areas.

WMAM had forecasted ambitious results in the current year, but in some respects they have not delivered on them. WMAM appears to be underachieving against their target in content pages at the midpoint of the current year, despite being on track to meet their targets in other areas. At the midpoint they have achieved just 10,404 Wikimedia pages of 173,340 forecasted.

This has resulted in some changes in the way that WMAM intends to assess their impact and performance. In particular, WMAM is shifting to a focus on quality content and on filling content gaps. Accordingly, they have adjusted their content-focused targets for the proposed year. Based on their achievements in the current year, it seems likely that they can meet these more modest targets.

Progress in participants and new editors indicates that WMAM is continuing their progress in mobilizing the Armenian public to do more work on Wikimedia Projects. The recognition of Western Armenian’s ISO code is an important achievement of this past year, which will benefit the long term development of both Eastern and Western Armenian Wikipedias and language projects.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

In past years, WMAM has done significant work to improve the way their organization is functioning, and to set up and clarify roles on their board. The organization’s quick assessment of the need for organizational change and the board’s ability to manage this transition, shows that these efforts have paid off. WMAM has onboarded a new Executive Director and is advancing further positive changes within the organization, to further strengthen this organization’s foundation. For example, the ED and staff have been specifically focused on streamlining WMAM's financial processes, which we pointed out were lacking last year, and settling their organizational structure.

We continue to appreciate the growing expertise of WMAM’s staff team.

The evident thoughtfulness in this proposal is a good indication that WMAM is a robust organization with the ability to assess and adapt to changing circumstances.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

The strategic plan updated this year works as a clear guide to implementation. While it gives a clear direction to the organization and outlines a foundation for WMAM's programs, it lacks clear alignment with the new movement strategic direction. We hope that WMAM will continue to update its strategic thinking, and with time integrate the themes within the movement strategic direction: knowledge equity and knowledge as a service. This year's proposal is full of thoughtful decisions and applied learning, and we are confident that WMAM is on track to participate in developing the movement strategy as well as to align with the strategic direction.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

WMAM has presented an improved program plan that showcases their strategic thinking and how they have learned from their past work. We like how their programs are each based on a clear theory of change and find that the activities proposed and the results sought are logically linked.

The shift from quantity to quality in WMAM’s approach to content is the very welcome outcome of a thorough assessment of the strengths and weaknesses of their past work, as well as their assessment of where the Armenian Wikimedia projects are developing. Their very lively and successful 'I create a vital article" campaign is likely to bridge content gaps and increase the quality of the Armenian language projects. This effort is especially well-supported by the creation of thematic councils and the engagement of experts to support the creation of missing vital articles. The digitization program works hand-in-hand with this to secure a better presence of Armenian content online.

Community engagement is an area where WMAM continuously shines, and the Community Program presented here is focused towards the motivation and empowerment of contributors, as well as capacity building. This is nicely designed to feed into their content-focused initiatives, so that both programs can work together to maximize impact. This work is done through editathons and more focused workshops and trainings, including technical skill building. WMAM is responding to the need to engage with contributors other than young contributors, and to change the way their work is perceived in Armenian society.

WMAM has been a leader in developing Education Program work in the Wikimedia Movement and is continuing to develop this area of their work by adjusting their approaches and embracing partnerships. WMAM is making some changes that will help streamline the administrative structure of WikiClubs and better integrate them with other program work. Their partnership with the Armenian General Benevolent Union (AGBU) will help grow their base and engage a more diverse public in WikiClubs to maximize their educational value. WikiCamps are also being revamped to include broader educational goals meant to better anchor Wikimedia work, which is building on a strength of the current program.

We are supportive of some of the major changes that WMAM intends to make in measuring and evaluating their results, including a shift to a focus on content quality, content gaps, and the educational value of their programs to achieve longer term societal change. Nevertheless, we do not yet see their evaluation plan fully reflecting this. For example, WMAM has been setting grants metrics targets for their programs at the activity level, but the metrics selected at this level are focused on granular targets such as events and participants, and are less likely to inform WMAM about the meaningful results of their work. Not only will that be a difficult evaluation plan to implement, but it is unlikely to result in relevant information that will inform us about how this programmatic shift is functioning.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

WMAM has responded to the need for change during the current year by reallocating resources in a timely and effective manner. This shows that their monitoring systems are working well. They are also in the process of improving aspects of their financial systems.

In the current plan, WMAM has narrowed the gap that was forecasted last year in staff compensation and have redirected resources to strengthen the organization and produce more impact. The new staffing plan and proposed use of resources shows WMAM investing in programs that are proven to lead to good results, while still adapting them to changing circumstances to keep them relevant and effective. Beyond direct online impact in the Wikimedia projects, the budget is focused on impact in their communities and society at large, with an aim to integrate Wikimedia culture at many different levels to achieve long term change.

WMAM has also made impressive progress in the current year in securing external funding.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	WMAM has a strategic plan in place. This proposal is effective at explaining how WMAM intends to implement their strategy and how and why they are changing their approaches in some areas based on their context.
P2. Potential for impact at scale	Strength	WMAM continues to operate programs on a national and international scale and to test out new approaches that can be adapted by others (this year, in the area of quality).
P3. Objectives and evaluation methods	Neither	While we recognize that improvements are being made, many of the metrics included here are not relevant enough. Including grantee-defined metrics at the activity level is likely to cause problems with the implementation of the evaluation plan.
P4. Diversity	Strength	Similar to last year, WMAM's communities already have a good gender balance and WMAM's work contributes to language and geographic diversity. We encourage WMAM to find ways to define diversity that are relevant in their environment and take proactive steps to address this.
Organizational effectiveness
O1. Past results	Strength	We appreciate continued successes in the areas of partnerships and movement building, yet not all quantitative results are on track at the midpoint. The large funding increase given last year should have led to improved results all around. The targets tied to the increase, especially in the realm of fundraising, are also not yet attained commensurate to the funding increase.
O2. Learning	Strength	We see that WMAM is documenting lessons learned in their proposal and targeting specific areas for improvement based on what they've learned.
O3. Improving movement practices	Strength	WMAM continues to collaborate with other movement groups, remains active in the CEE region, and is continuing to build and share movement practices.
O4. Community engagement	Major strength	This is an area where WMAM continues to shine. Community engagement is integral to all that they do, both in terms of the way the work and what they seek to achieve.
O5. Capacity	Major strength	We are impressed with the way WMAM's staff and board were able to handle a major transition in the organization. Board roles are functioning well and staff has significant expertise to offer. WMAM's activities continue to be supported by large numbers of dedicated volunteers.
Budget
B1. Past budgeting and spending	Neither	WMAM has made good progress in fundraising outside of the Wikimedia movement and has also been effective in monitoring their budget and adapting to new circumstances; nevertheless last year's increase was tied to certain areas of improvement that did not result in what was expected (for example, focused on ED salary and what they were going to achieve).
B2. Budget is focused on impact	Strength	This reduced budget should result in a more balanced allocation of resources.

This staff proposal assessment is the work of FDC staff and is submitted by: Delphine (WMF) (talk) 07:02, 15 May 2018 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.