Learning and Evaluation/Grants Impact/Fiscal Year 2013-14/Non-APG Grant Programs

This Grants Impact Analysis focuses on analyzing and summarizing the spending and outcomes of all grants funded in WMF Fiscal Year 2013-14, for the Project and Event Grants (PEG) and the Individual Engagement Grants (IEG) programs. Given this analysis coincided with a consultation on re-structuring the WMF grants programs, the "Analysis of Spending" was expanded to encompass across all grants programs except the Annual Plan Grants Program (i.e. PEG, IEG, TPS and Wikimania Scholarships).

This report differs from previous Grant Impact Analyses by focusing on all grants funded in Fiscal Year 2013-14, rather than grants for which a report was submitted in that fiscal year. While this does imply an overlap in grants analyzed, this report is not completely duplicative.

Executive Summary

$1,322,348 was granted in FY13-14 over all non-APG grants programs, 75% to PEG, 14% to IEG, and 11% combined across Travel and Participation Support (TPS) and Wikimania Scholarships.
- A total of 176 grants were awarded: 18 to IEG, 56 to PEG, 23 to TPG, and 78 Wikimania Scholarships.

The Proposed grants program structure leads to an approximately even distribution of total non-APG spending, across Projects, Conferences & Travel, and Simple Annual Plans
- Growth projects were a small portion of overall non-APG spending (10% of total non-APG spending; ~30% of total Project spending) in FY13-14
- Growth projects do not seem to cost more than Seed projects; 86% of FY13-14 Growth projects are under $11K

PEG and IEG grantee outcomes focused primarily on:
- Content, specifically adding new content to Wikimedia projects and gaining access to content by pursuing and cultivating partnerships.
- Participation, specifically understanding the contributor motivation and increasing the participation of contributors or volunteers.
- Quality, specifically improving the quality of current Wikimedia content and addressing content gaps.

Reporting of Global Metrics (before it was required for all grants) reveals varied levels of reporting, e.g. “Total Individuals involved” was reported in 70% of PEG & IEG grants, whereas “Absolute bytes” was reported in only 8% of grants.
- Only 62% of grants reported content through quantitative metrics (e.g. through the number of articles new/improved).

Recommendations to Community Resources and Learning & Evaluation

Additional reporting or follow-up may be necessary for tool building grants, as the current grant reporting period seems too short for grantees to observe and capture outcomes.
Offer Conference & Travel grants a different set of required metrics / outcomes, given content is not the focus of these grants (and therefore Global Metrics is only partially applicable).
- All FY13-14 PEG, IEG Conference and Travel grants seemed to have some component of the following: Experience sharing (incl. sharing and gaining knowledge), Networking (e.g. making contacts), Creating/starting new collaborations or projects.
Additional investigation of the following:
- The applicability and usefulness of “Absolute bytes”, to understand if it should be kept, removed, improved, or switched for a different quantitative metric.

The usefulness of the learning question on motivation, to understand if the question elicits and captures the intended information.
Outcomes tied to “Increasing awareness”, particularly to understand whether media mentions & social media following should be a quantitative measure of outreach and awareness.
Storytelling as an effective way to communicate outcomes in annual plans, as the activity-focused reporting format for annual plans does not seem effective in capturing outcomes.
The outcomes from the different types of partnerships pursued and established by grantees.

Overview of data and analysis

This report covers grants awarded in Wikimedia Foundation (WMF) fiscal year 2013-2014 (referred to as “FY13-14”), with a specific focus on the Project and Event Grant (PEG) program and the Individual Engagement Grant (IEG) program.
- A previous Grants Impact Analysis of PEG and IEG was done for all grants who submitted reports in FY13-14. While there is overlapping data between these two reports (i.e. grants that have been analyzed in both reports), the two reports do not cover the same set of grants.

At the time of this report, a new grants structure was under discussion. As such, analysis of spending was done by both the Current grants program structure (referred to as “Current structure”), and the Proposed grants program structure (referred to as “Proposed structure”).

For FY13-14, Global Metrics was not a required part of grantee reporting, and so the aggregated metrics in this report are likely lower due to under-reporting.
- Eight grants were either not complete (set to end in 2015) or did not have an “accepted” final report. For these grants, metrics and outcomes were aggregated from their most recent interim report, draft final report, or grant talk page discussion. Nine grants did not submit a final report.

Analysis of outcomes was meant as an exploration into grantee-reported qualitative outcomes, so how often an outcome was reported or the “number of mentions” is excluded from this report. Instead, quotes from grant reports are offered as examples of each qualitative outcome.

Analysis of Spending

Analysis of Spending includes all non-APG grants program spending, i.e. spending for PEG, IEG, TPS, and Wikimania Scholarships awarded in FY13-14.

Highlighted insight: Proposed grants program structure results in an even distribution of FY13-14 non-APG spending across Projects, Conferences & Travel (referred to as "Conf.+Travel" in the graph), and Simple Annual Plans.

Breakdown of WMF Fiscal Year 2013-14 grant spending (excluding the Annual Plan Grants program). Grant spending is broken down by current grant structure and proposed grant structure.

Other insights:

The majority of Project spending is on Seed projects, which are 70% of total spending & 70% of the total # of Project grants.
- Wiki Loves Monuments is ~44% of Growth Project spending (and ~50% of the total # of Growth projects)
A ~$30K spending threshold characterizes all Project grants, regardless if PEG or IEG, Seed or Growth projects
Most Project grants (across PEG and IEG) take 9-12 months to complete, even in IEG where the 6 month grant limit is a default.

Analysis of Outcomes

Analysis of Outcomes includes only PEG and IEG grants programs.

Overview of Outcomes

Grantees primarily reported outcomes in three areas: Content, Participation and Quality. Below are the six outcomes that characterized the majority of quantitative and qualitative outcomes reported, with quotes from PEG and IEG grant reports as illustration. While these three areas account for the majority of qualitative outcomes reported, it is not an exhaustive list; this analysis was meant as an exploration into understanding the most commonly reported areas of impact, rather then creating a comprehensive list of every outcome reported.

Content	Participation	Quality
Adding content to the Wikimedia projects by uploading images, creating/improving articles, etc.	Understanding motivation, i.e. why an individual chooses to participate or not	Improving the *quality of current* content**
	Some Wikipedians who self-identify as women may avoid certain parts of the project and/or certain kinds of work because they've experienced gender-based hostility in the past and/or don't want to engage in editing controversial articles. — IEG - "Women and Wikipedia"	Over 750 images have already been used to illustrate articles on at least 10 different language Wikis including English (Pakistan), Urdu (Karachi), Bengali (Badshahi Mosque), French (Derawar Fort); Italian (Badshahi Mosque) and German (Jahangir). — Wiki Loves Monuments, Pakistan 2014
Gaining access to content by pursuing and cultivating partnerships	Increasing participation, motivation and engagement	Addressing gaps in Wikimedia content
We have also made solid contact with several institutions which were previously hostile to the idea of uploading their institutional content to Wikimedia Commons but which are now looking for ways around bureaucratic hurdles so that they can upload their collections for the greater good. — Wikimedia US-DC, Projects in 2014	Mentorship is associated with increased productivity and retention of newer editors. — IEG - The Co-op	We have received a permission to publish the Bible in Ukrainian under a free license and are working on having it scanned. — Wikimedia Ukraine, 2013 Annual plan

Additional detail on Content outcomes

Total content reported added to Wikimedia Projects across PEG and IEG:


18,130 New or Improved Articles	4,664 pages, 66 books Added to Wikisource	249,332 Images or Media uploaded to Commons	7,776,827,316 Bytes Added

However, only 62% of PEG and IEG grants reported content. 18% of PEG and IEG grants did not report content, and these were typically grants for conferences, travel, research and tool building.

Additional investigation is needed to further understand the different types of partnerships that grantees pursue and establish, before outcomes can be summarized.

Additional detail on Participation outcomes

While only ~15% of grants reported lessons learned about “motivation”, three main lessons were reported:

Lesson learned #1: Hostility is demotivating	Lesson learned #2: Motivation is influenced by an individual’s personal interests and values	Lesson learned #3: Offline interaction and events can have a positive effect
What is most interesting to note is how the Gendergap mailing list networks changed between 2011 to 2014, specifically how overall participation decreased and the network became less diffuse. Several interviewees suggested that the Gendergap mailing list isn't a "safe space". — IEG - "Women and Wikipedia"	Wikipedians who self-identify as women may or may not prioritize research about and efforts to address the "gender gap." Individual factors, values, and the kinds of work women do influence the ways in which they interact with and conceive of Wikipedia's "gender gap." — IEG - "Women and Wikipedia"	I've proven my hypothesis that holding repeat events can build a social, effective community of female editors creating content to address systemic bias. I held social editing workshops at Loyola 7 times over 3 months… [with] 96% women per session… We created 72 articles. — IEG - Women Scientist Workshop Development

A total of 14,927 contributors and individuals were engaged through FY13-14 PEG and IEG grants. However, grantees reported increasing the participation, motivation or engagement of community members in four distinct ways:

#1: Attracting new contributors	#2: Retaining new contributors
There were 66...participants, who edit wiki for the first time, 18 of them continue editing after the camp... Students who already knew editing mastered their wiki knowledge and helped others. — Wikimedia Armenia, Wiki Camp 2014	Editors who were mentored made at least three times as many edits on average compared to editors who were not mentored. 68% of mentored editors remained active for one month after the pilot ended, whereas only 22% of non-mentored editors remained active in the same period. — IEG - The Co-op

#3: Re-activating inactive contributors	#4: Increasing the contribution of current contributors and volunteers
This project helped Telugu Wiki projects to… revive the participation of 2 existing wikipedians, who are not so active till this project started. — IEG - Making Telugu content accessible	While it is expected that not all of the project's volunteers would be further motivated, a small number of them have gone above and beyond the requirements of the project and have contributed to Wikipedia outside the scope of the project. — Wikimedia Philippines, Cultural Heritage Mapping project

Additional detail on Quality outcomes

Grantees reported improving the quality of current content both quantitatively and qualitatively.

Quantitative: Encompassed reporting articles, images, or media improved or awarded Good/Quality/Featured status	Qualitative: Involved a description of how Wikimedia content was improved
5399 Improved Articles 89 Awarded Articles 10,185 Unique images/media added to Wikimedia pages 241 Awarded media	For Wikipedia, [this project] allows increasing quality of wide range of articles by adding the relevant audio recordings to them, allowing the reader to "touch" the authentic audio material. — PEG - Kolessa recordings digitalisation

Outcomes related to addressing content gaps were only reported qualitatively, and in only ~10% of PEG and IEG grants overall.

The goal of [the NATO article contest] was to target content gaps about NATO in Ukrainian Wikipedia (only two dozens of articles) … the main reasons for this contest were… high interest of Ukrainian society to the topics related to NATO with low quality of articles about this topic in Ukrainian Wikipedia... This contest helped us get 315 articles [from 52 participants] about NATO...including 3 [Good Articles].
— Wikimedia Ukraine, 2014 Annual plan