Grants:APG/Proposals/2016-2017 round 2/Wikimédia France/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$603,852	$726,742	+20.35%
Budget	$1,150,498	$1,167,130	+1.45%
Staff (FTE)	11	11.5	+4.3%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• WMFR continues to develop their WikiMOOC program, which has the potential to reach a larger and more diverse audience.
• WMFR seeks to engage regions and local people by developing effective “Wiki Loves”-like activities that achieve large numbers of images in use.
• WMFR is supporting community health with trainings in non-violent communication.

Concerns

• WMFR’s achievements to date are not commensurate with past funding.
• WMFR is seeking to increase its grant amount from 570,000 € to 686,000 € but it seems unlikely that WMFR will achieve results which justify this increase.
• WMFR’s grantee-defined metrics do not capture the measurable impact of their work.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

French language projects may benefit and involve large numbers of people, in France and beyond.

While WMFR has opportunities for GLAM partnerships with renowned cultural institutions, the GLAM environment in France has presented an ongoing challenge for them and we have yet to see partnerships yield impact on the scale we would expect. We hope this improves in future years, as WMFR advances in this area.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

WMFR has integrated and reorganized their progress from the current year through the lens of this upcoming year's grant metrics, making it difficult to track and understand current progress against their objectives. They have provided high level aggregate metrics in their progress report, but it is often difficult to tie these achievements to particular program activities. It is also hard to track the coherence between results in the global overview and those reported and highlighted in the infographics or program highlights.

WMFR's progress report shows that they are not on track to meet their own targets for the current funding period. This could be due to poor performance, setting unrealistic targets, or a combination of these two factors. Yet, WMFR has set ambitious targets for participation and new editors for the next funding period, which they are unlikely to meet. The total of participants in WMFR's activities achieved at the 2015-16 midpoint is 1,759 out of a target of 18,440. While this result does not include some of the most participant-intensive activities (the MOOC and the WikiConcours Lycéen), it still does not seem likely that WMFR will meet their target by the end of the funding period.

WMFR’s target for pages was 185,380 in the current year, and they are not on track to achieve this result. WMFR reports 45,633 pages at the midpoint. WMFR reports 12,641 images uploaded by the midpoint, and that only 699 of these images have been reused on Wikimedia projects. However, elsewhere in the progress report they describe 32,918 images achieved through WLM, photography support, and Wikipédia l’Été des régions, but we do not know how many of these total images have been reused. So while WMFR is achieving large numbers of images uploaded, it seems like the use rate for images overall is inadequate or unreported.

The Wikipedia l’Été des régions project is producing exceptional results in terms of images in use, achieving 4,000 images used out of 15,000 total images since its creation. This might be because the program emphasizes using locally relevant images on locally relevant articles. This approach may be adopted by other Wikimedia groups with similar opportunities.

WMFR has supported 14 local groups (with about 100 participants overall) in France and support of local groups is an ongoing organizational focus. In the progress report, WMFR provides one example of how they have supported an event organized by two local groups, but the report does not contain information about how active these local groups may be. It is also unclear how the ongoing investment of WMFR’s staff and organizational resources is leading to results in that area.

The first WikiMOOC developed by WMFR has been successful in involving large numbers of participants (6,107 people registered, 699 completed) and may be raising general awareness about how to edit Wikimedia projects. We appreciate that it is run and curated by volunteer trainers. It is important that Wikimédia France builds on on its successes and continues to ensure that the number of people completing the MOOC stays high, and equally important that they refine their metrics and measurements in order to understand whether and how this program will lead to direct impact to the projects, like editor retention.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

WMFR plans to hire a full-time developer to work on the specifications of a platform for WikiMOOC that could be used by the Wikimedia movement, as well as for various technical needs from the organization (event websites, development of tools). While it is important that Wikimédia France's experience in that domain is taken into account, it is also important that they build on existing resources in the movement. This is a recurring concern as they seem to be developing new tools when it is not clear that they are considering existing ones (such as Pattypan for Commons work, or the Program and Events Dashboard, or existing MOOC-like platforms developed by other chapters like WMAR or WMIL). Managing a full-time developer will also be a challenge for the organization, since WMFR is not a technical organization.

WMFR is a well-resourced and well-staffed organization, and has had a large budget and staff for a number of years. Yet, year after year, results do not match the amount invested in those resources, while other less-resourced organizations taking part in the APG process have improved in their abilities to articulate their plans and strategies, and to achieve measurable results.

We are concerned to learn about the departure of several members of WMFR’s Board in past months, without public explanation. We do not know how this will affect the stability of the organization in the coming year.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

WMFR’s strategic plan (2014-2017) describes what the organization does in great detail. It seems to present an organizational framework rather than define a strategic direction. WMFR has plans to develop a new strategy, and is likely waiting on the results of the movement strategy process, yet it is concerning that an organization of this size is continuing to grow their reliance on APG funding without a cohesive and up-to-date strategic vision that justifies their ongoing maintenance and growth.

WMFR incorporates risk assessments and impact mapping to measure the outcomes for each set of activities described in their programs, which demonstrates expertise in planning their activities. However, their radars sometimes show that a planned activity will have minimal impact, or will not be sustainable. The rationale behind decisions leading to the implementation of activities is unclear, as well as how exactly these risk assessments affect those decisions.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

WMFR’s grantee-defined metrics in the upcoming year seek to measure the number of press mentions related to the organization, and the number of people trained as a result of their programs and events. These metrics are not sufficiently defined to offer relevant information about outcomes. For example, measuring the number of people trained without explicitly defining what that level means (Can they edit? Have they edited? Do they know that they can edit?) is unlikely to offer us any additional insight about the effectiveness of activities such as the WikiMOOC. The relevance of press mentions varies greatly with the circumstances, and may not reveal much about WMFR’s progress, especially since it is not clear what WMFR aims to achieve with the tracking of this metric.

Wikimédia France has described several interesting approaches to developing local activities and locally relevant content, building on past approaches in this area.

The idea of "Wiki Loves" such as l'Été des régions focuses on local heritage similar to Wiki Loves Monuments, but expands the concept to appeal to local communities eager to promote their regions by sharing pictures of traditional and local celebrations, sporting events or local arts and crafts. This takes advantage of local opportunities, such as the motivation of local participants, and the potential for local partnerships. WMFR has secured a partnership with the Association des maires ruraux de France (Association of French rural mayors), which will enable them to secure relevant local support for this action. The project "Ma commune Wikipédia" acts on the same principles.

Tools such as Lingua Libre and securing a partnership with Signes des Sens capture minority languages in the Wikimedia projects in an innovative way. Other organizations in the Wikimedia movement may learn from this work, and it is good to see the collaboration with Amical in this area.

We appreciate that WMFR is supporting its contributors by providing access to online resources, in response to the community’s expressed needs, and that WMFR is experimenting with training contributors in non-violent communication with the aim of creating a more inclusive environment for contributors. The impact of this work may be difficult to measure, but it is worthwhile.

WMFR is also continuing their work on WikiMOOC and supporting local groups. These approaches seem worthwhile, but we don’t yet have a good understanding of their potential for impact, since the achievements and targets provided do not adequately capture the outcomes of this work.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

Wikimedia France is requesting a large increase in the current year, although the FDC indicated last year that a 636,000€ grant was unlikely to lead to commensurate impact, and therefore chose to grant WMFR 571,000€—less funding than they requested.

WMFR is reliant on donations and APG funding to sustain their maintenance. Other APG organizations receiving very large grants are raising revenue from other diverse sources. While the overall growth of WMFR's budget from the current year to the proposed year is less than 2%, the 20% increase of their grant request is significant. In the upcoming funding period, WMFR forecasts an increased reliance on APG funding for revenue, despite having a reliable source of revenue through their own fundraising that is not available to many other chapters. This strategy runs counter to the FDC’s advice that WMFR seek to diversify its revenue rather than increase its reliance on APG funding.

A large portion of WMFR’s budget consists of expenses that are not strongly linked to impact. For example, Wikimédia France increased its travel costs by about 41% this year in comparison to last year's plan. (Their 2015-2016 plan lists 49,080€ in travel, while the 2016-2017 budget lists 85,280€.) There is no information in the proposal about why this increase is needed, for example whether travel funds are used for scholarships, members travel or staff travel. The amount of travel proportionally to the total budget is unusually high at 7.7%, and does not seem tied to direct impact.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	WMFR's strategic plan currently spans from 2014-2017, and a revised or adapted plan for the entire upcoming funding period is not yet available. It is good that WMFR has planned a new strategy process, but we would not expect a request for an increase in the upcoming year's grant amount without an up-to-date strategy.
P2. Potential for impact at scale	Neither	WMFR has good opportunities to work with French language projects that can benefit people around the world. WikiMOOC may have potential for impact at scale, if leveraged to adapt to target audiences. Past results do not bode well for achieving impact on the scale demanded at this level of funding.
P3. Objectives and evaluation methods	Concern	It is difficult to understand the reasons why Wikimédia France implements their programs and activities, or how their evaluation work influences the decisions they make about their program plans. WMFR has selected grantee-defined metrics that are unlikely to offer relevant information about their work. Furthermore, it is very difficult to link specific activities in this proposal to outcomes, as WMFR has provided targets at the aggregate level but has not offered SMART objectives at the program level or specific targets at the activity level.
P4. Diversity	Neither	We appreciate that WMFR is emphasizing work with minority languages such as sign language to contribute to linguistic diversity, and is exploring non-violent communication training that could lead to a more inclusive environment. However, WMFR should be doing more in the area of gender, given the level of funding they are requesting.
Organizational effectiveness
O1. Past results	Concern	Participation and content outcomes are low overall when compared with other APG organizations, and WMFR is not on track to meet their own past achievements. WMFR is achieving promising results for images in use through their local heritage program and is reaching a wide and diverse audience through WikiMOOC.
O2. Learning	Neither	Wikimédia France has presented detailed assessments of risks and opportunities with each of its programs. Yet its choice of activities does not always seem to map to obvious outcomes of these assessments. Some material included in these learning sections are also repetitive (for example, concerns about limited resources). Since WMFR is requesting a significant increase in funding, more is expected in this area.
O3. Improving movement practices	Strength	Wikimédia France continues to contribute to improving movement practices by sharing its work with the movement (working transparently on Meta, for example). It effectively engages with movement partners on various projects such as Lingua Libre (Amical and others), and Wiki Convention (francophone Wikimedia affiliates). WMFR also aims to develop a MOOC platform for all Wikimedia organizations. However, it is not clear how much Wikimédia France builds upon existing tools, thus taking advantage of existing movement resources.
O4. Community engagement	Neither	Wikimédia France seems to be struggling to find volunteers to sustain their activities. While the involvement of the editing volunteer community in the MOOC has been a success, and local groups seem to be doing well, it is not clear how engaged overall the editing community is with Wikimédia France.
O5. Capacity	Concern	While WMFR has significant staff capacity, it is not always clear what the nature of their support to various activities might be. There have been several unexplained departures from the Board in the past year. There are a large number of activities in this proposal and activities grouped as programs do not always seem closely related, which will make their plan difficult to implement.
Budget
B1. Past budgeting and spending	Concern	Results at the midpoint show they have not made progress in delivering impact commensurate to the level of funding received. WMFR also did not reorganize their budget or rethink their plans in response to the FDC's recommendations in 2015-2016 Round 2, although the FDC indicated they were not delivering impact on the scale of funds requested.
B2. Budget is focused on impact	Major concern	Based on the plan for the upcoming year and past results, it seems unlikely that the funding requested will lead to commensurate impact. WMFR's budget is not always focused on high-impact activities; for example, an unusually large portion of the budget is devoted to travel without ties to direct impact and WMFR has a large staff relative to the outcomes they are achieving.

This staff proposal assessment is the work of FDC staff and is submitted by: Morgan Jue (WMF) (talk) 18:59, 8 May 2017 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.