Grants:APG/Proposals/2017-2018 round 2/Wikimédia France/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$418,460	$549,000	31.19%
Budget	$1,289,186	$1,161,440	-9.91%
Staff (FTE)	8.0	8.0	0%

Overview

This section summarizes the themes that emerged from the proposal, which are explained in the detailed narrative section of this assessment.

Themes

• WMFR seems to be emerging from the crisis, but is still in a period of transition and uncertainty.
• The community rose to the challenge of resolving WMFR’s crisis, and leadership is listening actively to the community.
• Good work has been done to improve governance and realign with the movement.
• Evaluation needs to be strengthened and results need to better reflect the amount granted.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

WMFR has had to respond to many difficult events in the past year and while this has set them back in many ways, it has also restored many broken or lost relationships, with community, the movement and with institutional partners. WMFR’s turnaround seems to have led to a much more peaceful environment, and may have paved the way for fruitful collaborations in the future, as well as improved support of French language projects and the French community. Their first steps in listening to the community are demonstrated by a survey directed to WMFR's stakeholders, which is a good way to better understand and respond to their environment.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

Last year we pointed out that results were difficult to track in the case of WMFR. This year, WMFR has improved their ability to track results, but the results at midpoint are not on track to meet targets set in the last proposal. While we note that WMFR has already exceeded their target for content pages added or improved at the midpoint of the current year (31,348 for a forecast of 16,700), other indicators are lacking granularity that would allow us to know whether or not they will succeed.

It seems that a lot of the activities involving participants and new editors will happen in the second half of the grant period and this means that numbers at the midpoint look low compared with what WMFR will actually achieve this year. WMFR’s achievement for total participants is at 1,319, compared with a target of 9,965, and the number of new editors is at 178 against a target of 2,720. Moreover, results and forecasts are hardly consistent from year to year, making it difficult to track how results are relevant to the work done. However, it seems WMFR is tackling this problem now and we may see improvements in this area in the future.

The targets set in this proposal are ambitious compared with the results achieved in the previous year (Fiscal year 2016-2017) or the midpoint results this current year (FY 2017-2018). For example, WMFR is expecting to achieve a result of 14,075 participants this grant period, while it has achieved 1,759 in all of fiscal year 2016-2017 and 1,319 at the midpoint for fiscal year 2017-2018. WMFR still needs to improve in its ability to forecast their impact more accurately and learn from past results and achievements. We welcome the change in grantee-defined metrics. Volunteer hours and partnership satisfaction will probably paint a much more interesting picture of WMFR's work than the current year’s press mentions and people trained grantee-defined metrics. Setting up a performance report should improve the traceability of results (understanding where the results are coming from), and this is a good step forward by WMFR.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

WMFR has spent the last six months redesigning and stabilizing their organizational structure. The effectiveness of WMFR has been questioned in the past. The plan devised this year, with fewer employees (from 11 to 8) and clearer mandates for the board, the staff, and even some community members (for example, through work on the the microgrant committee) seems to show that a thoughtful change is happening.

For the time being, WMFR has decided to forego a traditional executive structure with an ED. WMFR has spent a lot of time assessing their own capacity and skills, including the capacity and skills of board and staff, to come up with the design they have adopted. They also have conducted a governance audit which will probably bring insights for what they need to look out for in the future of their organization. This new structure is well suited to their current staffing, but will need to stand the test of time.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

WMFR has had a strategic plan in place for the period 2014-2017 which was due to be reviewed and revised. Because of the circumstances of 2017, this did not happen. This proposal and the work WMFR has done to reorganize themselves indicates that strategy is important to the organization. It will be interesting to see how WMFR develops their next plan. This work is supposed to start now, in parallel with the movement strategy process. We look forward to observing how WMFR will address alignment with the new strategy direction.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

In last year’s staff assessment we observed that the grantee-defined metrics selected by WMFR (number of press mentions and number of people trained) lacked clarity and relevance. These metrics have been replaced by number of volunteer hours and partner satisfaction. The number of volunteer hours metric might be a good indicator of community engagement in the work of the chapter, and after the crisis WMFR experienced it is all the more relevant to understand how the community finds ways to engage and get involved. The relevance of the partner satisfaction criteria is less clear, especially since the definition of "partner" is not yet well developed, but seems to align with WMFR’s aim of renewing and strengthening relationships with its partners.

WMFR's program structure has evolved from last year, with a stronger focus on education and training rather than on advocacy. This plan includes a thoughtful approach to redirecting resources to more impactful areas of work. In the content improvement program, joining l'Été des régions with Ma Commune Wikipédia seems to be a good way to build upon interesting programs that aim to cover content gaps in heritage articles.

WMFR also has revisited its approach to GLAM partnerships and refocused its efforts to change its dynamics with partners from an ad hoc relationship that manifested through sporadic events to one with more structure and deep relationship patterns. Their GLAM strategy hinges on the creation of a GLAM network that will help organizations work together more effectively in the GLAM sphere. WMFR benefits from high profile partnerships (Centre National d'Études Spatiales, Archives Nationales) and is working on innovative partnerships such as their "Wikimuseum" campaign with the Palais des Beaux-Arts de Lille.

WMFR’s approach to re-engaging with local and regional communities is likely to be successful. We see this in their desire to evaluate how WMFR and communities can interact and work together most effectively, and understand the needs of local communities. This is a good step toward enhancing and showcasing the work of communities while simultaneously reevaluating the real needs that WMFR can fulfill.

WMFR has included a number of activities reflecting a growing interest in addressing diversity: Lingua Libre with the introduction of a tool for oral knowledge; dedicated gender gap initiatives with relevant third party partnerships, and the creation of a Women and gender working group; the collaboration with Signes de sens for the development of sign language content; and minority languages initiatives. However, this plan lacks focused metrics that measure the specific outcomes of these diversity-related activities. We encourage WMFR to develop a set of metrics designed to give a more concrete picture of the impact of diversity-related initiatives, similar to, for example, the metric "Contribution activities to reduce the gender gap on Wikimedia projects" (under "Working with communities").

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

WMFR presents a drastic reduction of its budget in this plan, breaking the trend that the FDC has pointed to in recent years of an organization that was growing without commensurate results. This plan's budget at 952,000 € for the coming year is lower than the budget forecasted for the current year (at 1,056,710 €). While this is the result of a crisis, we appreciate that WMFR is rethinking their budget and staffing model completely, which may lead to a much more concentrated use of resources on actual impact. The hiring of a fundraising position may also help the organization optimize its potential for funding outside the movement.

This reduced budget and the redesign of the organization, programs and staffing plan, leads us to conclude that the potential for impact is much higher than it was in the plan we reviewed last year. WMFR is still the organization asking for the greatest amount of funding in the FDC process, despite not being focused on work with emerging communities.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	We encourage WMFR to rethink their long term strategy in light of ongoing movement strategy discussions, and the many changes that this organization has gone through.
P2. Potential for impact at scale	Neither	WMFR has an opportunity to increase their potential for impact at scale, as they are reexamining their organization and the way it is working. It will be important for them to find ways to adequately measure this.
P3. Objectives and evaluation methods	Neither	We see that this area is already improving as a result of changes in this organization. For example, the metrics included in this proposal are more relevant and it is much easier to understand why WMFR is doing the work they are doing.
P4. Diversity	Neither	While we appreciate that WMFR has integrated the idea of diversity throughout their programs and has emphasized this in their plan, we note that WMFR is not measuring lots of results in their gender work. We appreciate the way WMFR looks at different aspects of diversity, for example through their work on spoken language and sign language.
Organizational effectiveness
O1. Past results	Concern	WMFR is surpassing their 2015-2016 achievements in most areas, but are not on track to meet their 2016-2017 targets at the midpoint. Past problems with the structure of WMFR's program plan and evaluation plan have made this area difficult to evaluate. We look forward to improvements in future years, now that the organization has restructured.
O2. Learning	Strength	WMFR has shown that they are able to learn from past challenges and radically change their approach when that is needed.
O3. Improving movement practices	Neither	As WMFR transitions, we hope to see them play an active role in the international movement, as well as take advantage of relationships with their peers to improve their own programs.
O4. Community engagement	Strength	We appreciate the role that the community has played in revitalizing and restructuring WMFR, and the responsiveness of the organization's new leadership to community voices.
O5. Capacity	Neither	While this organization is going through a transitional phase that may be difficult to manage, we appreciate the work that has been done on the governance audit and in the restructuring process. The important reduction of their staffing model is a good step in rethinking the organization's capacity and effectiveness.
Budget
B1. Past budgeting and spending	Concern	Past budgeting and spending was difficult to link to direct impact. WMFR's new leadership has worked on new allocations for the budget 2017-2018 and while the transition has cost the organization a lot, it seems that prudent spending is on track.
B2. Budget is focused on impact	Neither	We believe the changes to the organization's structure and budget will better align to the impact they will achieve in future years.

This staff proposal assessment is the work of FDC staff and is submitted by: Delphine (WMF) (talk) 07:05, 15 May 2018 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.