Grants talk:IdeaLab/Full Circle Gap Protocol: Addressing the Unknown Unknowns

Handling potential, negative responses to "feminist" content

First, I very much like the idea of tapping into one-time and/or sporadic editing to address gaps in content! One question: Given Wikipedia's NPOV policy and how some community members respond to content they perceive as not being "encyclopedia worthy" (e.g., being politically motivated, irrelevant, etc.) how will you handle and/or train your facilitators to handle these kinds of questions? --Mssemantics (talk) 16:16, 17 March 2015 (UTC)

@Mssemantics: Thanks so much for the feedback! In this pilot, undergraduate students will be trained to edit Wikipedia using the 'gap lists' generated by expert advisers. This partnership is really ideal for addressing issues around NPOV since the Wikipedia Education Foundation has training materials designed specifically to ensure that students will be able to engage in meaningful ways. Of course, I anticipate there will be hiccups along the way, particularly since we'll be trying something new with respect to the creative way that content gaps will be identified. But we have the benefit of time, students will spend a quarter (10 weeks) learning and editing. Shameran81 (talk) 17:27, 19 March 2015 (UTC)

initial thoughts and feedback

@Shameran81: very interested by your idea and I love the approach of pairing with the Wikipedia Education Program and leveraging some of the existing infrastructure and know-how from the program. I'd also love to see more details about this work... who are the feminist academics you propose to engage? Have they already been introduced to the idea and expressed interest in being involved? Also, would there be opportunities to bring in feminist scholars (or even non-scholars for that matter) who are outside of the University of Washington to assist with generative ideas around de-biasing Wikipedia and processes related to identification and review of content? -Thepwnco (talk) 01:08, 18 March 2015 (UTC)

@Thepwnco: Hello -- thanks for the feedback! This proposal specifies having the pilot "brainstorming" event with ~five UW scholars (a loose category that includes graduate students, librarians, researchers, instructors, and professors) in part because of the interest generated last month among participants at a critical, feminist edit-a-thon I co-hosted at UW: "I Love To You: Critical Wikipedia Edit-a-thon". It's worth mentioning that this proposal, and the partnership with Wikipedia Education Program, is a direct result of the collaborative energy and conversation that took place at I Love To You. No one has been formally "invited" yet to be an adviser and participate in the brainstorming event proposed here, but the provisional plan is to invite the five advisers from this group, given their existing interest and expertise. However, there is absolutely room for flexibility in recruitment, especially if this IdeaLab proposal generates lots of interest among scholars in the Seattle area who may not have been involved in the I Love To You event. The goal of the pilot project is, of course, to see if and how this slightly different outreach model might be useful and interesting for others and for Wikipedia. Shameran81 (talk) 17:28, 19 March 2015 (UTC)

Care must be taken in selecting article subjects

We all know that there are hundreds of thousands of women who meet notability criteria on various Wikipedias but do not have articles. However, in some cases the potential article subjects *prefer* not to have articles about them. There are good reasons for this: it dramatically increases their public profile in a way that they cannot control; little-watched and little-read articles are more likely to be "hijacked" by POV-pushers and either vandalized or written in a biased manner (whether positive or negative); sometimes an initial article is perceived as "too commercial" and then is deleted with a summary about "not advertising" or a deletion discussion disparages the article as spam/astroturfing, whether or not this is correct. These all have the potential to have a negative effect on the on-line profile of the article subject. So yes, identifying gaps is very important. But care has to be taken to ensure that we are not causing harm by creating Wikipedia articles that create more problems than they solve. Many women carefully curate their on-line profile to protect themselves from real-world physical or psychological harm. Risker (talk) 17:44, 19 March 2015 (UTC)

@Risker: Thanks, and yes, I couldn't agree more. In fact, while certainly there may be biographical entries on Wikipedia of important historical and contemporary living persons that are incomplete or missing, writing up biographies of these women -- particularly living, breathing women -- as the solution to content bias is misguided at best. But, in my experience at edit-a-thons, the writing of bios happens because (sometimes new) editors, inspired by critiques of bias on Wikipedia and looking to help, are actually not sure where to start. While I don't have any solid evidence for this, I'd venture that writing a biography might seem like it's just a relatively straightforward way to improve to the encyclopedia in an afternoon writing session. But as you rightly point out, such an approach can be harmful. I also would add that biographies do not get to the gut of systemic bias. The approach proposed here aims to get closer to doing that -- respectfully tapping the expertise of feminist scholars to have them read between the lines of existing content and create annotated lists of sites to improve, this approach can also serve as a more grounded starting point for future editors to make meaningful contributions that go beyond biographies. Shameran81 (talk) 17:22, 20 March 2015 (UTC)

@Risker: I've been thinking about my response to your question. I'm afraid that by seeking to highlight what is distinct about my approach, I come across as dismissive of efforts to write biographies. I regret this. I appreciate and support the folks and projects that seek to write, and edit, biographies. My project idea may take a different tact, but certainly that's not to say it won't have anything to do with biographies -- I actually don't know that yet -- nor that there is not more than one route to inclusiveness -- there are (and should be) many and multiple approaches. Shameran81 (talk) 04:19, 25 March 2015 (UTC)

Making subject matter expert's (SME) edits "stick"

One thing that all academics/SMEs need to understand, not just feminist ones, is that their contributions to WP can be easily reverted by anyone at anytime. So, there needs to be a plan included in the proposal to ensure that the edits/inputs to WP articles "stick" and frankly, the only way to do that, is to get a number of people to watch the articles constantly and indefinitely and revert any attempts to remove the SME's edits. There's no other way around it. It's how WP works. I suggest that the proposal include language to recruit students or others to place the articles edited by the SMEs on their watchlists so they can immediately address any attempts to remove or alter the SME's edits. The students can use the Wikiproject:Feminism talk page to coordinate their efforts to protect these articles once they've been improved by the feminist SMEs. Cla68 (talk) 23:50, 25 March 2015 (UTC)

@Cla68: Thanks for taking time to read and comment on this proposal. The question of how edits and contributions "stick" is interesting and important, particularly because your comment foregrounds how much labor goes into policing the encyclopedia! To start out, we will have the task of following up/watching the contributions as they live online as part of editor training with undergraduates. Coordinating with WikiProjects could be a great addition to this process. Shameran81 (talk) 19:33, 28 March 2015 (UTC)

Tapping WikiProjects

Since I assume this project is targeted to English Wikipedia (at least initially), it would be good to invite the participation of the women-related WikiProjects there:

• WikiProject Feminism
• WikiProject Women's History
• WikiProject Women scientists
• WikiProject Women writers
• WikiProject Women of psychology
• WikiProject Women's sport
• WikiProject Women artists
• WikiProject Countering systemic bias

These projects usually have a good knowledge of what is lacking as far as missing and undeveloped articles related to their particular subject areas. Many of them have already done significant work to build lists of such gaps. It would be great to combine forces with them. Kaldari (talk) 01:28, 26 March 2015 (UTC)

@Kaldari: Thanks! I've already reached out to a few, definitely can tap more. Possibly, as @Cla68: suggested above, some WikiProject members may be interested in involvement as guardians or watchguards of the new content. Shameran81 (talk) 19:33, 28 March 2015 (UTC)

Eligibility confirmed, Inspire Campaign

This Inspire Grant proposal is under review!

We've confirmed your proposal is eligible for the Inspire Campaign review. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period.

The committee's formal review begins on 6 April 2015, and grants will be announced at the end of April. See the schedule for more details.

Questions? Contact us at grants(at)wikimedia.org.

Proposal feedback

Hi Shameran81 and Ragesoss. Thanks so much for this interesting proposal. We're excited about the model you're piloting and it's potential for scale. We are in the final stages of review and have the following remaining questions:

1. The creation of a documented workflow for engaging subject matter experts who identify content gaps that then feed into education program work lists is what makes this project impactful in the long run. It would be great to add this to the measures of success.

   I think that's something we had planned to do anyway; I'll add it to the measures of success.--Ragesoss (talk) 17:59, 27 April 2015 (UTC)

   Yes, thanks for adding that detail. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

2. We would like to see content creation metrics added to the measures of success. For example, quantitative goals around creating/improving articles (adding references, creating categories, cleaning up articles to remove systemic bias, or uploading better images).

   I added a conservative content creation goal, which is basically a minimum threshold for a class having taken a real crack at addressing identified gaps. Hopefully the numbers should end up quite a bit higher, but the goal is around the numbers I'd look for to see whether the concept was worth trying to scale up. (User:Shameran81, feel free to overwrite what I've done there.)--Ragesoss (talk) 18:42, 27 April 2015 (UTC)

   I think that is a fair estimate, even a bit conservative. Thanks for including it User:Ragesoss. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

3. We assume that the content gaps lists created by the experts will be available to anyone who wants to see them/work off of them. Please confirm.

   Yes, definitely.--Ragesoss (talk) 17:59, 27 April 2015 (UTC)

   Yes! --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

4. $200 for refreshments for 5 expert reviewers seems on the high side. We're comfortable keeping that budget as is, but with the hopes that you'll be able to recruit a few more experts. If the budget is not used fully, remaining funds can be returned.

   Sound reasonable to me.--Ragesoss (talk) 17:59, 27 April 2015 (UTC)

   In fact, the number is actually for refreshments for two events, though I see now this detail isn't explicit in the proposal. Light refreshments will be served at the gap finding event and at the final class presentation session. During the class presentation, the subject matter experts will join to meet the students who have taken the task of editing, then stay on to give a final evaluation and reflection. The estimate covers both events. I'll clarify this in the proposal. Of course, remaining funds from all of the categories can and will be returned. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

5. Generally, we hope that subject matter experts would participate in this type of activity in a volunteer role. However, if we will defer to your best judgement if this is the case and you need to offer honorariums to successfully recruit experts.

   The idea here is that we're asking quite a bit more from these folks than would be asked from a regular expert participant in a gap finding activity; we want them to not only do the gap finding activity, but also help us think through and refine the protocol. From my perspective, at least, I don't think the honorarium is something we'd expect to include except during this protocol development project, but in this context, it's important for signalling our respect for these experts and their time.--Ragesoss (talk) 17:59, 27 April 2015 (UTC)

   I'll chime in here too. Yes, it's absolutely possible that the subject matter experts would volunteer for this, the honorarium is a one-time recognition that they're involved in the development of the protocol, a commitment that spans across six months. This signals respect for their involvement and while it may not be a full compensation for their time, it could -- along with the childcare provision -- more fully enable participation. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

6. Can you please provide a few more specifics on the responsibilities of the Project Manager to help us better understand the 10hrs/week estimate?

   As the project manager, I will be responsible for the day-to-day managing, anticipating, forecasting, reflecting, controlling, communicating, organizing, delegating, evaluating, and documenting of the project. Having this job is crucial to the success of the project since the aim is not only to organize events (itself a big job with lots of moving parts) but to keep thoughtful documentation of the gap-finding workflow process in order to write the protocol. The hourly time is an estimated weekly breakdown of possible total hours -- with the idea that some weeks may necessitate a heavier workload than others. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

7. It's great that you're partnering with the Wikipedia Education Foundation. Have you thought about how this project could tie into the National Women's Studies Association (NWSA) Wikipedia Initiative?

   (With my Wiki Ed hat on now) Wiki Ed is excited about this — along with a somewhat similar gap finding project in a Global Feminist Art course going on right now — as a proof of concept for gap-finding as a scalable mode of contribution from academia. We'll use our contact network from our NWSA partnership to try to recruit courses for the second phase of this project (filling the gaps). Beyond that, it will depend on how well it goes; if it goes well, then it's something we would probably incorporate into many of our partnerships down the road. (cc User:Jami (Wiki Ed), please correct me if I've overstepped here.)--Sage (Wiki Ed) (talk) 18:05, 27 April 2015 (UTC)

   The National Women's Association NWSA Wikipedia Initiative is fantastic! In fact, I look forward to learning more myself about it. :) I agree that the gap finding model may well be very useful for future initiative beyond a Wiki ED partnership. As we develop the partnership, the evaluation and future recommendations can include specific recommendations. --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

Looking forward to your responses. Cheers, Alex Wang (WMF) (talk) 17:46, 27 April 2015 (UTC)

Thanks User:AWang (WMF) for taking the time to thoughtfully respond and ask questions about the proposal. It's been a generative pleasure to work with User:Ragesoss and Wiki Ed so far and I am very excited about this going forward. Let me know if there's anything else to address or help with. All best! --Shameran81 (talk) 16:20, 28 April 2015 (UTC)

Aggregated feedback from the committee for Full Circle Gap Protocol: Addressing the ‘Unknown Unknowns’

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	8.6
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	8.4
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	8.9
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	7.9
Additional comments from the Committee: This project seeks to improve gender diversity in content by systematically identifying gaps in the knowledge represented on Wikipedia and developing key resources that can support contributors in filling in the gaps. Has potential for online impact, especially with a focus on creating resources for others to easily adapt and use in their own events/projects. Success with this plan could also be used in other projects. Like that it is focusing on content, a smart approach that could be replicated. Really interesting project that could potentially address one of the less well-known aspects of the gender gap, and that it could have a high potential positive impact across the Wikimedia movement. Community engagement includes participants from previous Wikipedia edit-a-thon held by the grantee, plus students at the grantee's university. Also nice to see support from various parties, including WEP, Art+Feminism, and the grantee's university research commons Appreciate the participants have engaged well with the community throughout the proposal process. Project is very well-scoped. The metrics for success seem generally reasonable, and it seems like they could easily be shared across the Wikimedia movement in order to carry out similar projects focused on different target communities. Might like to see metric to measure the number of articles created and the number of instructors and students. Consider adding content creation as a measure of success with goals set, perhaps quantitative goals of adding refs, creating or expanding articles, creating categories, or cleaning up articles to remove systemic bias, or uploading better images. Like that the participants have included the often-overlooked cost of PM. The budget seems very realistic, and the organizers seem to have the necessary mixture of skills and experience to execute the project in the proposed timeframe. It might be nice to see something about the development/testing of scalable workflow protocol for similar events that could be run on other topics Like how the proposal recognizes the burden/challenges faced by marginalized people and is realistic about not expecting everyone to become into a long-term contributor. Appreciate that the work of identifying where Wikipedia's knowledge gaps are is important, and challenging - informatics measures like alt-metrics (http://www.altmetric.com/blog/new-source-alert-wikipedia/) might make it possible to compare Wikipedia's coverage of knowledge relative to other publishers and platforms. This is an excellent, reasonably-scaled proposal focusing on one of the less well known aspects of the gender gap.

Inspire funding decision

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $7000

Comments regarding this decision:
Thanks for engaging in the Inspire campaign! We’ll be in touch about setup soon.

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.

Experts selection

On «feminist scholars affiliated with the University of Washington -- these are the people who are capable of recognizing the ‘unknown unknowns’ of gender-related content --», are you saying that nobody beyond said scholars in said university is capable of such a thing? --Nemo 15:33, 6 September 2015 (UTC)

Why events

Why did you choose events as a system to produce lists of topics? Is that really more effective than a systematic approach? Traditional encyclopedias and dictionaries have existed for some centuries before Wikipedia, surely the methods to build them have not been entirely forgotten yet. Did you review existing sources, such as those included in Mix'n'match, and the methods used to build them? For instance, I used lists by BEIC and ANPI to promote creation of relevant articles on women. --Nemo 15:33, 6 September 2015 (UTC)

Conclusion: indeed it worked poorly. I can understand just 1 article created, but 40 students involved (and no mention of how many were women?) is really little. Usually we'd invest few hundreds dollars for a class like this. Nemo 13:35, 3 June 2016 (UTC)