Wikimedia monthly activities meetings/Quarterly reviews/Grantmaking/September 2014

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Grantmaking team, September 29, 2014, 11:00AM - 12:30PM PDT.

Present (in the office): Lila Tretikov, Erik Möller, Damon Sicore, Garfield Byrd, Anasuya Sengupta, Toby Negrin, Siko Bouterse, Floor Koudijs, Alex Wang, Winifred Olliff, Katy Love, Edward Galvez, Haitham Shammaa, Anna Koval (taking minutes), , Maria Cruz (taking minutes), Jaime Anstee, Kacie Harold, Janice Tud, Marti Johnson

Participating remotely: Asaf Bartov, Sati Houston, Jonathan Morgan, Tighe Flanagan, Jake Orlowitz

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Agenda

Overview
Highlights: Q1 progress, Q2 goals
- Diversity strategy
- Global Metrics
- Annual Plan Grants
- IdeaLab
Key needs

Overview

Anasuya: Our team directly supports community members with grants and non-monetary support. We've supported 4,000 people directly and 42,000 people indirectly in the last year.
Lila: 4000 people got grants?
Anasuya: Yes, either grants or resources like a Wikipedia Library subscription. This is the multiplier effect in terms of impact and outcomes. We're reaching people on many different projects and in many different languages. But the real focus for us is the Global South. Since August 2012, GN participation is diminshing, while GS is slowly increasing.

Slide 4

Anasuya: We look at languages, including English. India is the largest contributor from the GS on EN. 1:4 contributors in Indonesia is contributing in English.
Lila: Next time overlap percent of GS active vs EN.
Anasuya: Quality is still an issue. We have very little understanding of quality. The one thing we know is that diversity is a key enabler of quality.

Slide 5

Lila: Is this just editing?
Haitham: These are number of created articles. Not deleted.
Lila: Do we have our deletion rate for the same period. The GN looks terrible.
Toby: Why are we just looking at EN?
Anasuya: we look at enwiki as the easiest way currently of looking at systemic bias. English, in the GS, is seen as the place to be by many. We analyze other languages separately.
Lila: We can bucket it like that but we need to look at both.
Siko: GS research is an ongoing area of focus that is not complete for Q1, this is some early data.
Anauya: Asaf will walk you through other projects we want to focus on. We want to be able to tell both stories.
Siko: Contributors from other language WPs will tell that EN is often their gateway into those languages.
Lila: do we have data on how they find WP?
Toby: Eric Zachte has done research cross wiki as well.
Anasuya: Jonathan is going to be looking at entry points.
Lila: We need to understand motivators. Are you working with the language team?
Anasuya: On the GS survey, we are working cross-team with language engineering, mobile, Wikipedia Zero...
Lila: Toby, do you have somebody proofing the final copy so we don't have bias? Before we push it out, it should be validated by the data team.

Slide 6

Anasuya: Gender has the same bias. In some ways it is incredibly frustrating, because it is not showing any changes. WMF hasn't done anything consistent about it, and that is what we want to do now.

Slide 7

Anasuya: Jonathan has been working on looking at the content gap. Categories around female and male.
Lila: What is the x axis?
Anasuya: Different language Wikipedias and their articles tagged with male or female categories. 29% of articles in TL.WP are tagged female. EN is at about 18%. It tells us very clearly what the systemic bias is. I want to put this into context: in the next few years we'll be looking at GS from contributor potential and content gaps.

Slide 8

Siko: Context for knowing the diversity of the GS. that's one of our key goals for this year. 1. Focus. 2. Evaluation is another core for us: both identifying what works, and also creating metrics that are significant to us.
3. Integration. Reorganization of our department and its structures. Non-monetary support, tools, guidance and mentorship.

Slide 9

Siko: Q1 progress: Focus: GS, gender gap studies. Evaluation: global metrics launched. Integration: IdeaLab and grants programs pipeline. Most of the grants we see went to the GS.
Lila: Are we just looking at GS grants? Are they just individual grants?
Asaf: There are no APG grants made in Q1.
Anasuya: There are no FDC grants made this early in the year, which is why we have pretty balanced numbers in terms of GS/GN diversity.
Damon: What is FDC?
Siko: Large grants to large Wikimedia organizations.

Slide 10

Siko: We are just going to focus details Q1 discussion on: (1) Diversity; (2) Global South; (3) Global Metrics (4) IdeaLab

Slide 12

Asaf: In Q1, we turned the corner on catalyst programs that have turned into partnership grants. India is continuing. Brazil is not continuing. There was a successful community consultation in Brazil. We are planning the same model of community consultation in India. There are separate communities in India, and significantly harder to get together, compared to other contexts. We also worked with WM Ukraine and WM Serbia on leadership transition.
In Q2/Q3, we'll be establishing baselines through the GS survey, but we also want to understand the specific norms and community health and culture and dynamics and expectations of different GS communities. Based on that we will create a baseline off-wiki activity plan, and encourage all GS groups, whatever their form of self-organization, to adopt this plan as a first tier of programmatic goals. Finally, we'll develop a portfolio of partnership models to address the content gap about GS content, and to increase our reach in terms of offline efforts.
Lila: I think there is a lot of meaningful work here. I would like to see more precision about where we're trying to get. Focus on how we measure our impact. Who the grantees are. If we were to focus on 5 languages, what would this look like? Let's put a straw man together.
Asaf: What we have is a process that will result in concrete moves. We don't have numbers to attach to this. Some of these communities do not have existing grants, and I am tasked to support them whether they have grants or not.
Lila: What does success look like when this process is over?
Asaf: We derive our support for these communities from the shared vision. First we have to get the expectations clear about roles and responsibilities.
Anasuya: You are working towards a customized goal with each editor community.
Lila: I'm trying to identify where we should be doubling down on our efforts and where we should pull back and restructure.
Asaf: we are doing the opposite of doubling down in some communities, like Brazil. We stopped the catalyst program, but not the support. We are drawing these conclusions right now.
Lila: I'm trying to get clarity on outcomes.
Anasuya: We are focusing on 9-10 different geographies, doing a needs assessment of what they want to do and what we are able to do.

Slide 13

Siko: The gender gap is something the movement and the foundation has talked a lot about but have not put resources behind. We're crossing our fingers that this is the year to change that. Seven strategic initiatives: that's the Q1 baseline. We would like to carry this down two tracks: supporting developing teams (some of them will be funded, others may just need people's time and attention); the other track is the Inspire campaign.
Lila: Is 7 too many?
Siko: I, too, wonder about that. There will likely be a culling process. We are nothing if not ambitious. The gender gap is the area where we would like to test the Inspire campaign, proactively seeking projects aimed at the gender gap. WMF can't lead in direct content-creation campaigns, we have to be careful not to have a negative effect on communities. But we can support the communities and teams that want to execute on local content campaigns.
Lila: How are you approaching that? Content?
Siko: We can support communities to plan and launch content campaigns, as well as other projects aimed at the gender gap. Ideas like admin training, etc, have already come up from the community, and we can do more to support community health projects too, not just content campaigns.
Lila: Admin Campaigns? Female Admins?
Siko: We are well placed to support the community in those regards. We are not well placed to do that work ourselves, of course.
Anasuya: Along the lines of WLM and periodic competitions, we are hoping that the Inspire campaign, if successful could become a seasonal drive
Siko: Aiming to run the first pilot in March, building on WikiWomen's History Month and other events w/ International Women's Day.
Lila: Will need to think who is the audience? Are they current Wikipedians? Or are they new - we would need to do market research to understand motivation, for instance, if that was the case. One of the things I think about with the female audience, is if we try to create new communities, or if we are trying to pivot the current community.
Lila: Should work with Product on it if you aim for a new audience, rather than existing community. It goes hand in hand with Product. It is product thinking.
Siko: Agreed.
Lila: Do you create new sub-communities which are a majority women? or do you pivot the main community to be more accepting? We need to be very careful how we think about that. Would like to dig in deeper there as you work on this.

Slide 15

Jonathan: (introduces Global Metrics). We expect these metrics to give us a high level idea of what impact programs and projects have online. The learning question aims to understand what is their motivation. So far, we have focused on developing resources to help program and project leaders understand this new tool (webinars, learning patterns, office hours). We identified these metrics and then a set of resources to help the community understand them. Got feedback from grantees. This isn't meant to be a comprehensive set of every way you could evaluate projects.
Lila: when you were piloting the metrics with grantees, how did that work out?
Jonathan: We have just begun the pilot period.
Anasuya: I feel really proud of our team here, because it could have been a disaster. The notion that we actually want to see aggregate impact in this way could have been a disaster. We were very careful deploying this facing the community, making sure we emphasized this is only one way of understanding impact. We socialized this concept in different ways, and when we finally launched the metrics, it had a lot of buy-in. The process has been on for over 6 months.
Lila: What will the pilot look like?
Anasuya: We can watch PEG first, and APG will start to report on those metrics next year.
Lila: What is your goal for the end of the year?
Jaime: Target is 125% increase of reporting this year compared to last year's baseline. We essentially have the baseline of how these metrics have be reported and now will wait to see the change.
Lila: How many grantees reported last year?
Anasuya: we have about 95% reporting from all our grantees. What we didn't have until now was systematic metrics.
Lila: Do you expect all of them to be consistent this year?
Anasuya: What we found was that focus determines design.
Lila: What are you trying to get at?
Anasuya: We want community members to focus on specific results.
Lila: We should have it somewhere so it's clear. Out of the 6, do we have infrastructure to track them?
Anasuya: Wikimetrics!
Jaime: some of them are covered with Wikimetrics, others need tools like Glamorous, and surveys.
Anasuya: We didn't include readership, because we don't have good ways to measure it.
Toby: These other tools, besides Wikimetrics, seem important too. They're things like Magnus Manske's special projects right now. And they could use some support.
Anasuya: This is our point: that tools like this need support.
Siko: Piloting this year will help us understand how to measure these metrics best, and also show larger patterns in gaps in tools and infrastructure for measurement.
Toby: My team has been focusing on Wikimetrics, as part of our work on other tools as well. We don't have a dedicated team on that specific tool though.
Erik (to Damon): Wikimetrics is a cohort analysis tool.

Slide 17

Katy: The TL;DR version is we are streamlining our backend processes (streamlined proposal; lighter reporting requirements). We are now seeing much improved reporting on outputs and outcomes, though not with all our grantees. Let's remember that two years ago there were no grants, no reporting requirements. This fall, a lot of support has gone into helping grant applicants write SMART objectives. We are seeing increased impact. As the slide shows, in the first two quarters of this year, we are seeing our grantees reach 50% more participants than they did in total 2012-2013. Total participants is an indicator of improvement, as well as pageviews, article creation. For instance: grants to WM Serbia and WM Amical triped in size last year, and we're watching to see if they triple their impacts, and we're so far seeing that happen. WMRS for instance has quadrupled the number of articles and Amical has tripled their participation rates--just in the first two quarters. We're also very interested in policy/advocacy work, since our grantees have a special role that they could play in that arena. However, we don't have good metrics on policy and advocacy work.
Lila: We have to be careful around that.
Katy: We agree. We are working with the legal team on what we can help them with and what we can't. What they are most well positioned to do.
Anasuya: What the education team is working on in terms of policy is much easier to track.
Katy: We really want to see an increase in grantee impact. We want organizations to find support in us not only in the financial sense, but also in other areas. We want to help them become more impactful.
Lila: Do all APG grants go through FDC?
Siko: Most do, but not all.
Anasuya: Some chapters we don't want to transition to APGs because they don't need full time staffing.
Katy: We have hands on approaches with all of them. Wikimedia Ukraine, for example, we're encouraging to stay with PEG.
Lila: what is our outcome strategy? Do we want everybody to be on multiple PEGs?
Anasuya: Part of it is to make sure that our organizations focus on what they can do well, and don't grow too quickly if not necessary.
Lila: We have huge leverage here. We can incentivize behavior. What behavior do we want to incentivize? Behavior or outcome?
Katy: That is the back end. We don't want grantees to be thinking on what type of grant they are need. They just need funds to execute a program. We need them to focus on impact to achieve movement goals.

Slide 19

Siko: IdeaLab is a place where we source ideas and support the community.
We invest in community building around our programs. We have grantees, we have reviewers, and then we have community commenters. They show up and either comment on proposals or endorse them. They help us scale the work around review and mentorship. They help us make some decisions in terms of the funds we grant. We will never have the capacity to understand if projects or programs will work in every given given context, so building a community is very useful in this sense, because they are, in a way, our eyes in that context. Here we see reasons for community motivation to participate in IEGs.

Slide 20

Siko: They are interested in the ideas and impact, and that is one of the most valuable aspects of this project.
We wanted to make it easy to start an idea and build a community around them. To this end, we focused on FormWizard, which is like our visual editor for creating proposals. We started building this for IdeaLab but did so in such a way that it could be used across our grant programs, and we're looking at expanding it further. We added 2 other tools: the ProBox (a Lua-based infobox), and an AddMe gadget.
Lila: suppose someone has an idea, how do they get to this place?
Siko: We wish we had better tools to bring people from their home-wikis to Meta. Without Global Notifications, it is difficult. We use Special:MassMessage to reach Village Pumps. We've experimented with CentralNotice to only reach logged in folks and it didn't go well (lots of readers are logged in, so not targeted enough).
Lila: can we target the central notice for registered users with three or more edits?
Erik: we could do that.
Anasuya: That would be a nice strategy.
Lila: Those are single digits. This participation is not even statistically significant based on the size of our community. If you look at it as a funnel of interest, you are not getting enough critical mass.
Siko: We all agree and would like to be able to work more broadly.
Lila: You may not be able to do so in Q2 but at least you understand your own limitations.
Siko: Yes, but one of the bottlenecks is of people not knowing how to get involved when they get there, and that's something we can solve w/o Engineering, so we've focused on things like AddMe first.
Lila: It's a problem with the system. How many people are we losing at each step in the process?
Siko: Once folks get in there, what do they do? We've focused on easy workflows so people can identify what a project needs and have easy ways to get involved.
Lila: Can I find a project from here, if I'm a researcher?
Siko: That was the dream. At this point, we have built half of that and we do not have resources to build the other half. We are under-staffed for projects like this.
Anasuya: We've talked about it being a matchmaking space, where community members find each other, matching ideas with resources, even beyond grants.
Siko: What we have committed to working on in the coming quarter. We know that we want to use IdeaLab as part of the Inspire campaign. We want to start measuring the impact of these improvements, but that will require backfilling some open staff positions as well.
Anasuya: It's a great segue.

Slide 22

Anasuya: Key needs. They are both needs, as well as challenges and opportunities. We want to redesign our team structure to focus on these goals we highlighted. We do need from you is to sign off on this proposal. The second piece is staffing. We are really really stretched on the team. We need 2 grants officers immediately to support Individual Grants, the Inspire Campaign. In any community engagement process, you think about not only what you need now, but that the future could bring really great ideas, and have way more demand. We really have been thinking of what a content-related strategy means to us. We see gender gap, global south. Someone who holds that will be very important going forward. Geocoded data: we've been discussing this with Toby for the past year. We hope that geocoded data will help us identify existing communities, and work further to empower them, and make a more meaningful impact. be better and be bigger. We would love to have integration with product and engineering to do that.

(end of slide deck and Grantmaking presentation)

Feedback

Lila: I think we're doing too much and spreading ourselves too thin. Instead of focusing on 20 things, if you could focus on 5, what would you focus on. I think we need to go deeper not wider. I don't think there is capacity to do it all. We need to simplify and go deeper in a few aspects. The one thing I have not seen here is looking at was is happening at the ecosystem, and planning it proactively. Is a GLAM program looking out, as opposed to having people come and find us? Who are the editors we need to be engaging? We have enough data to figure out who are the top contributors in the past 30 days. It's a manual process, but we can get at that. [Lila later realized she had not yet seen the focus strategy doc the team had shared with her which addresses many of these issues]
Anasuya: Those are two great pieces of a puzzle. The proactive vs reactive focus piece is complicated in that we are community facing. At this point in time, there are a lot of expectations of our team. We don't want communities to think we have left them behind. So we have to be careful in being seen as only proactive.
Lila: It's setting appropriate expectations?
Anasuya: We have improved in the past two years in who we reach out to, for instance, with Brazil and India, we've been reaching out to both offline contributors and the top online contributors. We are making sure to reach out. Siko, for example scans the ecosystem for new individual projects regularly. The Wikipedia Library came this way.
Lila: TWL should be more integrated.
Anasuya: We've just brought Jake on.
Lila: Tools integration is important.
Asaf: We have been doing this kind of proactive outreach, as Anasuya just said. We've also been doing community mapping. Lila, on what basis are you so certain that we should do less? How do you know that it's too much. Do you think you know our activities well enough to make that judgment?
Lila: This is an observation, not a judgment. I'm seeing how many people we have on staff, how many projects we engage on, and what each one demands.
Asaf: When we're over-extended, doing less is one way to solve it; adding resources is another way.
Lila: You want to add resources when you can't scale results? I'm not telling you what you need to do. I'm telling you when you come to these meetings you need to have a prioritization: here are the must haves, here are the wish to haves.
Asaf: and is scaling results the only criterion you accept for things we do? That is, are there no conceivable things we might need to do for the movement as stewards or custodians that may not make the top of the list in terms of scalable results?
Anasuya: We have been balancing proactive with responsive. But there is a must-have about the responsive.
Lila: I'm not saying the proactive is not happening, I'm just saying it didn't come out today.
Erik: On the technology and product side, we need to think about highly specialized solutions for highly specialized programs vs highly generalized tools like Wikimetrics. And making available new data sets. For highly specialized tools, we should partner with program leaders in the community to have them develop these tools. For the first category of integrations, Wikimetrics got a ton of support. So there is a lot of alignment already with Grantmaking needs. When it comes to things like deep product integration, we want to have a team in the Foundation that is focused more closely with the communities.
Anasuya: As we are. If we are moving to a much more proactive structure, we are going to need much more tech support internally. There needs to be a larger long term strategy around that.
Lila: it should show success and then Product can invest. We need to integrate these projects in the communities. Let's say the library is a good one, someone in product needs to look at it and see what is the threshold of success and how much staffing do we need so that we can match it. And it seems like Growth may be the place to evaluate these things.
Erik: We also need to look at your team's short term needs. Like I did on Friday with Frank Schulenburg and Floor with regard to the education program's needs.
Lila: I think the next steps is to group about this and determine next steps.