Grants:APG/Proposals/2015-2016 round 2/Wikimédia France/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$588,988.80	$654,432	18%
Budget	$1,184,521.92	$1,184,521.92	0%
Staff	9.5	11	16%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• Wikimédia France is documenting its approaches and results, and applying learning to improve its results
• Focused program plan with emphasis on online impact
• Evidence of increased volunteer involvement in program activities

Concerns

• Grant increase is unlikely to lead to commensurate increase in impact
• Proposed targets do not appear to be feasible based on past results
• Past results are not commensurate with past grant amounts

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• In last year’s assessment, we noted that Wikimédia France is continuing to assess opportunities in its local context for forming high-potential partnerships, and has many opportunities to do this within France. WMFR has continued to deepen partnerships with highly reputable national organizations. We appreciate that WMFR continues to learn from its partnerships, and continually surveys partner organizations.
• We also noted that as a result of its location in Paris, Wikimédia France is well-positioned to work on French language projects, including French Wikipedia, which have significant potential to disseminate knowledge globally, and that there are opportunities for Wikimédia France to identify and fill content gaps on French-language projects. We see that Wikimédia France is taking advantage of this opportunity through its pursuit of local content and knowledge initiatives.
• Finally, we encouraged Wikimédia France to continue to include information about how volunteers will be involved in each of its programs. In its proposal, Wikimédia France has emphasized its online engagement strategies and described how it is leveraging volunteer needs and interests to make its work more effective.
• Due to its focus on French language projects, Wikimédia France may have opportunities to extend its impact beyond France. Wikimédia France is building partnerships that it hopes will expand its reach in geographic areas where there are few active community members. Through scalable initiatives like WikiMOOC, Wikimédia France may achieve impact on French Wikipedia outside of France.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

Wikimédia France has achieved some important successes in influencing Freedom of Panorama legislation, with positive outcomes that could have long term impact on the legal environment in France. We also appreciate the detailed documentation around this process included in the progress report and the associated learning pattern.

In its most recent progress report, Wikimédia France has made an effort to show cumulative results of some of its longer term initiatives. We appreciate the views metrics provided for some of the midpoint results. For example, for the partnership with the National Archives, nearly 300 documents were uploaded over 20 months and viewed 30 million times. We also appreciate the innovative nature of initiatives such as the partnership with the CREM (Ethnomusicology Research Centre) CNRS–Musée de l‘Homme, the MMSH (Maison mediterranéenne des sciences de l’homme) sound archives. We also appreciate the detailed reports of how WMFR engaged with the community with each program. Also in the current year, Wikimédia France launched WikiMOOC, a massive online course aimed at educating people about how to contribute to Wikipedia, involving 2,000 participants so far.

Yet, overall past performance does not indicate that Wikimédia France is likely to achieve the targets included in this proposal in the areas of: articles created or improved or images in use, new editors involved, active editors involved, or individuals involved.

In the proposal, Wikimédia France targets 31,020 images in use compared with the current year’s achievement of 4,324 at the midpoint, and a FY 2014-2015 achievement of 5,057 images in use according to the FY 2014-2015 impact report. For articles created or improved, proposed targets are even more ambitious, with a target of 35,320 in the current proposal, compared with 311 achieved so far in the current year and 614 achieved in FY 2014-2015. This is in a year where Wikimedia France expects its overall budget to remain stable at 1,086,000 Euro.

It is difficult to evaluate Wikimédia France’s progress against its own goals in terms of images used or articles improved, since it did not set targets in these areas in the current year; however, in the area of participation (new editors and total individuals involved), it is not on track to meet its targets. Wikimédia France is exceeding its achievements from FY 2014-2015 in these areas; however, the current year’s midpoint achievement in active editors is less than what most organizations in Round 1 had achieved at their last year’s midpoint (with the exception of Wikimedia Israel and Wikimedia UK). Achievements in images used were higher than what any other organization except Wikimedia Deutschland had achieved at last year’s midpoint (although Wikimédia France’s grant amount is at least twice that of most other organizations in the APG process). Wikimédia France has the lowest midpoint article achievement of any APG organization.

While some of Wikimédia France’s achievements are not captured by global metrics, these metrics indicate that Wikimédia France seems to lag behind other APG organizations in the areas of participation and content, despite significant grant amounts year over year.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

Last year, we noted the successful transition of Wikimédia France’s executive director. This year, we note that the organization’s staff and board leadership continue to be stable and effective.

We greatly appreciate how Wikimédia France articulates the needs and requests of the online communities it works with throughout this proposal. This has been important to our understanding of Wikimédia France’s engagement with online communities, with an increased appreciation for Wikimédia France’s capacity. We hope that this approach continues to pay off, with volunteer-led activities and a close connection between the chapter and the communities.

Wikimédia France is a learning organization. This proposal explains the learning context behind each major activity well, including a description of what went well and what needed to be changed in order to improve performance. Beyond documenting program successes and challenges, it is nice to see this documentation of how exactly programs are being adapted in response to learning. We have enjoyed understanding how these programs and activities have evolved over time in response to learning. Wikimédia France continues to lead in this area.

Furthermore, Wikimédia France’s approach to learning shines through in its reports and program documentation. For example, Wikimédia France frequently integrates learning patterns into its reporting. Given the innovative nature of many of its initiatives, this is especially useful. Wikimédia France seeks to refine ways it can serve its partners by regularly measuring partner satisfaction.

In last year’s assessment we noted our concerns around a lack of external funding, and note that this has been an ongoing challenge in the current year.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan? An organization of this size needs a more highly developed strategy, and a way to track progress against long term strategic goals. We are looking forward to Wikimédia France’s new strategic plan, which will be developed in the current year, as we think Wikimédia France has potential to show leadership in this area.

The organization of the proposal has improved significantly this year, with an emphasis on online impact and community engagement. Programs were clearly presented, and logic models helped highlight longer term outcomes targeted through Wikimédia France’s programs. Content, Regional, and Advocacy work are well-aligned with online impact, while the International program and Tools work is less well-aligned (refer to more detailed concerns in the Programs section of this assessment).

We note that Wikimédia France plans to build its communications strategy in the coming year, and we look forward to the results.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

We are impressed with many of Wikimédia France’s innovative programs and approaches.

We encourage Wikimédia France to build on the success of its Freedom of Panorama advocacy work, and look forward to future work in this area. Working toward more self-sustaining partnerships through the development of the ComeOn! Tool has helped partners upload content with the appropriate metadata and licensing information. We appreciate that Wikimédia France has considered volunteer interests when determining thematic areas to pursue. On the other hand, it would be good to better understand if this thematic content actually fills gaps identified by contributors or readers. It will be interesting to monitor the use rates around thematic content in articles, in order to better understand how the thematic content is being used.

WikiMOOC, which has the potential to reach larger and diverse audiences (including women and people in Africa). We encourage Wikimédia France to coordinate with other movement organizations developing online platforms if they see themselves as a leader in this area. Local knowledge approaches, such as the Lingua Libre tool and the project around documenting local musical instruments, as well as activities targeting local expertise, may bring valuable content to the Wikimedia projects that could be useful globally. We appreciate that Wikimédia is building on past expertise with its work with sound and video content.

However, we have some concerns about work we think will be less effective.

Wikimédia France is continuing its programming in Africa, despite concerns about impact and local involvement. This is heavily emphasized as a part of its international program: while Wikimédia France includes the French Wikiconvention in its international work, most participants will come from France. Money spent on sending Wikipedians to international meetings may not result in commensurate impact (e.g. 50,000 Euro on scholarships).

Support to photographers, continued through WMFR’s tools program, targets 20,000 files, but without understanding how these media files are being used, it is difficult to assess the value of this program. Relevant results from the contributor spaces program are not clearly articulated. It also does not seem like the Open Data Symposium is likely to result in commensurate impact, and it may not be strategically aligned. Participation in conferences and symposia and educational videos project should have clearer goals.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

We are concerned that Wikimédia France is requesting an 18% increase in its grant amount, a significant increase from 540,000 euros granted last year to 636,000 euros requested this year.

Upon request from FDC staff, Wikimédia France supplied a detailed budget which included expenses by line item as well as by program. As we noted in last year’s staff assessment, we appreciate Wikimédia France’s competence in the area of producing detailed budget documents. At the same time, we have noted significant discrepancies in planned spending per program reflected in the impact report for FY 2014-2015 across many of the axes/programs. FY 2013-2014 Wikimédia France has moved significant funding from program expenses to staff and operating expenses.

Staff and operational costs make up a significant portion of the overall budget. For consistency with other organizations, we have included Program 6 as operating costs, since it is primarily focused on fundraising and communications. By this calculation, programs 1-5 make up only15% of the total budget. It should be noted that significant amounts of staff time are spent on and budgeted per program, so overall program spending would be a much larger proportion of the overall budget if staff costs are taken into account. Large amounts are devoted to scholarships (50K Euro) and printing (about 25K Euro), which may not lead to online impact.

At nearly 168,000 Euro, program and staff expenses for WMFR’s regional work make up a significant portion of Wikimédia France’s budget at about 15% of the total budget (this higher cost is due to a significant investment of staff time); at 140,000 Euro, program and staff expenses for Increasing content make up about 13% of the total budget (the bulk of this cost is in program expenses that include scholarships rather than staff time). International work, Tools, and Environment represent smaller portions of program and staff costs, at 8%, 5%, and 8% of total costs respectively. This means that program and staff costs are weighted toward impactful work like the regional and content-focused programs.

Yet, based on past performance and the plan proposed, we do not think it is likely that the significant increase in grant amount from 540,000 Euro to 636,000 Euro will lead to commensurate impact. This also would lead to an increased reliance on APG funding compared with the current year, from 50% of total funding from APG to 59% of total funding from APG. More external funding is needed to maintain or grow Wikimédia France’s current operations.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Neither	The current year's proposal is greatly improved from previous years, as it includes detailed logic models, clear explanations of learning, and a coherent program structure. We are encouraged that the plan focuses on online impact. On the other hand, the organization currently has a minimal strategic plan in place. We look forward to the development of WMFR's new strategic plan in the coming year, as we see WMFR developing leadership in this area.
P2. Potential for impact at scale	Neither	The amount requested at 636K EUR is very large, and it will be challenging for WMFR to achieve impact at a scale commensurate with this level of funding. We do see high potential programs that could become scalable approaches, such as the WikiMOOC program and WMFR's approach to developing the ComeOn! tool to help partnerships become more self-sustaining, as well as their approach to incubating local groups and locally relevant content themes. At the same time, we note that the ambitious targets in this proposal do not seem realistic based on WMFR's current and past achievements. For example, WMFR is targeting 31,020 images in use and has achieved 4,324 at the midpoint of the current year. WMFR is targeting 35,320 articles next year and yet has achieved 311 in the current year. Finally, WMFR is targeting 18,480 individuals involved next year, and has achieved 3,783 by the midpoint of the current year.
P3. Objectives and evaluation methods	Strength	This year's simplified plan will allow the organization to measure results and ensure learning. The proposal clearly articulates how the organization is using learning to modify its programs. By using global metrics and setting clear targets, WMFR employs effective evaluation tools and systems. Programs include SMART objectives, targets, and detailed logic models which articulate outputs and outcomes. As WMFR continues its work in the area of advocacy, we look forward to learning how the organization plans to track the long term changes achieved through this work.
P4. Diversity	Neither	The impact of most of WMFR's work is likely to be focused in France, although WikiMOOC may expand the participation in and reach of the Wikimedia movement beyond France's borders. Gender is not a major focus of WMFR's plan.
Organizational effectiveness
O1. Past results	Concern	Despite significant grants in the past, this organization has not produced results commensurate with funding. For example, WMFR received a grant of 600K EUR for the year 2014-15 which resulted in 5,000 images in use and 614 articles. In the current year, they've achieved 4,300 images in use and just 311 articles with a grant of 540K EUR. In the current year (FY 2015-2016), WMFR did not set targets for images in use or articles, since WMFR was using a different evaluation framework.
O2. Learning	Major strength	WMFR is a leader in this area. WMFR is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs. This approach is evident in the proposal, as well as in reports and through the creation of quality learning patterns. The organization has been consistent about documenting its failure, which enables themselves and others to learn from their work.
O3. Improving movement practices	Strength	At this level of requested funding, expectations in this area are high. This organization effectively shares learning about its work with the broader movement and beyond, on mailing lists, Meta pages and through additional resources (e.g. Local Guide, Partnership Framework, Wikicheese). WMFR wants to help others in the movement achieve more impact, although it would be interesting to more closely track how this WMFR's support to other organizations is leading to results in order to better understand the impact of this work.
O4. Community engagement	Strength	WMFR has improved its focus on engaging with its online communities, including conducting a community survey. The proposal clearly outlines how volunteer and community input is applied for each program. Major activities like WikiMOOC and Wikiconvention will be led by volunteers, showing continued engagement with offline volunteers as well.
O5. Capacity	Neither	WMFR has a committed board and staff. At the same time we have concerns about the feasibility of the plan and the targets included, given the data we have about WMFR's past performance. WMFR has been improving its engagement strategies with volunteers online and offline, which will provide a good foundation for future capacity. At the same time, WMFR has not had success bringing in significant external funding to support its growth, which should be a key component of diversifying resources and building capacity. While we see many strengths in the area of capacity, it is unclear why this has not led to results commensurate with the size of the grants WMFR has received in the past.
Budget
B1. Past budgeting and spending	Neither	WMFR has a mixed track record of budgeting and managing funds. While we are grateful for the detailed budgets WMFR has provided in the past, we note that spending within program areas has not been on track for the past few years. WMFR has a history of moving funds from program expenses to staff and operating expenses, and spending inconsistently within program areas. Finally, despite receiving significant grants each year, we have not seen results commensurate with the funding invested.
B2. Budget is focused on impact	Major concern	Based on past performance and the proposed plans, funds are not always allocated to programs and activities with corresponding potential for impact. Significant funding is allocated to printing, scholarships, and operating costs. On the other hand, program plus staff costs are weighted towards territorial and content programs, which have good potential for impact. Overall, WMFR has requested a significant increase in its grant that is unlikely to lead to results sufficient to justify this increase, which leads to a major concern in this area. Last year WMFR's grant was 540K EUR and this year the organization has requested an 18% increase.

This staff proposal assessment is the work of FDC staff and is submitted by: KLove (WMF) (talk) 23:34, 9 May 2016 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O3. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O4. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O5. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O6. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.