This project is funded by an Individual Engagement Grant

Welcome to this project's final report! This report shares the outcomes, impact and learnings from the Individual Engagement Grantee's 6-month project.

Part 1: The Project

Note: In the following report, the researcher, Amanda Menking, has chosen to use first person pronouns because the work described below is largely ethnographic and she acknowledges herself, her perspectives, and her lived experiences as a part of the research lens and methodology.

Summary

This journalistic choice [regarding whom to interview about Categorygate] demonstrates that there was a predetermined narrative set up: sexist men on Wikipedia need to be tutored by feminist women outside it. Thus the voices of women on Wikipedia or feminist Wikipedians in the academy would have troubled that story. The narrative told by the media replicates some of the same problems that they are identifying with Wikipedia. By refusing to let the voices of the minority be heard - female Wikipedians - they are reinscribing and replicating an easy narrative about how sexism can simply be fixed by “adding women” to Wikipedia. Even if this narrative sounds feminist and progressive - it isn’t. By ignoring all of the women already on Wikipedia, it is as if we are all invisible. As if all of the contributions, hard work, and debates we already contribute are utterly disregarded. Also, assuming that the women of Wikipedia haven’t thoughtfully considered many of these difficult questions about female representation and categorization, raised them on the site, and wrestled with them is insulting and, frankly, disheartening to the many of us that are here. In fact, the project to counter systemic bias was created on Wikipedia way back in October 2004.
— Adrianne Wadewitz, HASTAC

My project was intended to investigate the "gender gap" on the English language Wikipedia. My initial research questions were:

Who are the women of Wikipedia?
What motivates them to contribute?
How do they perceive the "gender gap"?

I wanted to collect and then provide the Wikimedia community with aggregate qualitative and quantitative data that could be used to assess existing efforts to address the gender gap. My project goals were:

To elicit and collect the stories of active editors who self-identify as women and identify central themes around their motivations, the kinds of work they do, and their views of the "gender gap." (Completed.)
To participant in and observe co-located and non co-located editing and mentoring activities. (Completed.)
To measure the success of co-located and non co-located editing and mentoring activities. (Incomplete.)
To analyze the effectiveness of community outreach in the case of the "gender gap." (In progress.)
To aggregate data from the community, sharing suggestions for interventions and the design of research kits for exploring the "gender gap" in non-US, non-English communities. (In progress.)

In the end, I did not meet all of my project goals. However, I collected and analyzed data that led to theory development and a new way of understanding the "gender gap." I also collected data that I'm happy to provide to Wikipedia communities and/or other researchers so that this work may continue:

Survey data
Preliminary, qualitative analysis of Gendergap mailing list and tools for others to continue analysis
Social Network Analysis of Gendergap mailing list

Contributions

Please note: Qualitative research is not intended to provide generalizable results but rather is done to understand a particular phenomenon and/or ground the researcher in the data so that he/she/they may then proceed with informed quantitative work. During this IEG, I employed member checking, peer debriefing, bracketing, and balance to help ensure validity.

Who are the women of Wikipedia?

Although my interviews focussed on editors who self-identify as women and edit the English language Wikipedia, I spoke with contributors from six countries. While each individual is an individual and intersectionality is an essential consideration, a grounded theory analysis of the first 20 interview transcripts supports the following:

The "women of Wikipedia" represent a diverse range of ages, relationship statuses, gender identities, life stages, languages, and cultural backgrounds.
Only one of the participants I interviewed currently has young children (e.g., infant to elementary age). Several of the interviewees currently have teenage or adult children; the remainder do not have children.
Wikipedians who self-identify as women are likely to be well-educated (e.g., to be pursuing or have attained at least a post-secondary education; several participants are pursuing or have attained graduate and post-graduate degrees). This finding is in keeping with other studies that suggest the average education level of Wikipedians skews up.
Some Wikipedians who self-identify as women may avoid certain parts of the project and/or certain kinds of work because they've experienced gender-based hostility in the past and/or don't want to engage in editing controversial articles.

What motivates them to contribute?

Many participants articulated a sense of "usefulness" as a motivation to contribute.
Several participants expressed motivations associated with personal learning/knowledge acquisition.
A few participants said they were motivated to contribute in direct response to the "gender gap."
On-going research asks whether, in some instances, statements of "ideological gain" may signal "deep acting" and mask emotional labor.

How do they perceive the "gender gap"?

Wikipedians who self-identify as women may or may not prioritize research about and efforts to address the "gender gap." Individual factors, values, and the kinds of work women do influence the ways in which they interact with and conceive of Wikipedia's "gender gap."

Please note: The findings above confirm Sarah Stierch's 2011 report: Women and Wikipedia Survey 2011.

Emotional Labor/Emotion Work

And part of me thinks that it [the gender gap] shouldn’t—not that it [the gender gap] shouldn’t be fixed because I really think that this community would benefit from more women in it and those are voices that need to be heard. But I also don’t want to drag other women down into what has [...] become a cesspool. [...] I keep talking about how awful Wikipedia is—and there are bright spots and I have plunged myself into those bright spots—but if—there’s a lot of nastiness. And I don’t want to drag other women down into that.
— Interview participant

Applying and extending the theoretical lens of emotional labor to the work that some women (and, though outside the scope of this project, doubtless some men) engage in so that they may contribute to Wikipedia is the most interesting contribution of the work supported by this IEG.

When any editor feels he/she/they must respond to hostility in a way that undermines or creates dissonance between how he/she/they really feel and the kinds of responses the community has normalized, he/she/they may be required to participate in a kind of "surface acting." Several interviewees reported these norms as being expressed through stereotypical masculine language (for example, "man up," "toughen up," "don't act like a girl"). These community norms may act as "feeling rules."

This note explores the issue of women’s participation in Wikipedia through the lens of emotional labor. Using a grounded theory approach, we detail the kinds of tasks women Wikipedians choose to do and explore why they choose the work they do. We also explore the emotional costs of their labor and their strategies for coping. Our analysis of 20 interviews leads us to posit that the gendered and emotional labor required of many women to participate in Wikipedia’s production renders it, problematically, a space of conflicting public and private spheres, motivated by antithetical open and closed values. In addition to other contributions, we believe this insight sheds light on some of the complex dynamics behind Wikipedia’s observed gender gap.

Abstract from "The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's Largest Online Encyclopedia"

Methods and activities

Research

Interviews (N=33)

The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's Largest Online Encyclopedia

Survey (N=90*)

Raw survey data.

Gendergap mailing list social network analysis and preliminary coding

Social Network Analysis - These images were created using Gephi and are also available in other formats. (Many thanks to David McDonald.) The size of the node represents the degree centrality, and the darkness of the shade of each color represents the betweenness centrality. What is most interesting to note is how the Gendergap mailing list networks changed between 2011 to 2014, specifically how overall participation decreased and the network became less diffuse. Several interviewees suggested that the Gendergap mailing list isn't a "safe space." Future research might further investigate not only the network dynamics but the kinds of participation that take place on the mailing list with the goal of suggesting interventions to make the list a safer space--if that is, indeed, the goal of the list.

Figure 1. Social network analysis of Gendergap mailing list, 2011

Figure 2. Social network analysis of Gendergap mailing list, 2012

Figure 3. Social network analysis of Gendergap mailing list, 2013

Figure 4. Social network analysis of Gendergap mailing list, 2014

Preliminary qualitative coding of nine months of Gendergap mailing list threads - We selected three months from the beginning of the mailing list, three from the middle, and three from the end (of data collection) to analyze.

Focus groups (N=3)

These transcripts and notes will be used for a future publication.

Community Engagement

I Love To You: Critical Edit-a-thon (2/14/2015)

Mentoring graduate and undergraduate students (for 2 quarters)

Anonymized reflections from students

Posts (2) for the Wikimedia Foundation Blog

Sharing findings with participants via email
Participating in Wikimania 2014 workshop activities
Upcoming presentation at CH1 2015 (engagement with broader research community)
Inspire Grants campaign funding committee member
Advising other researchers re: the "gender gap" and how to engage thoughtfully with Wikipedians and Wikipedia communities

Outcomes and impact

Outcomes

My research generated more questions than answers. However, I believe I've contributed to a new way to frame the "gender gap" and have made data available to Wikipedia communities and/or other researchers so that they may continue this work.

Progress towards stated goals

Planned measure of success (include numeric target, if applicable)	Actual result	Explanation
Aggregated qualitative and quantitative data re: success of sponsored versus unsponsored events with regard to user edits and longevity of users	Incomplete	My scope was too large, and my focus shifted. I was unable to complete this analysis.
10-15 interviews with women who edit Wikipedia	In total, I have interviewed 28 Wikipedians who self-identify as women. I have also interviewed 5 editors who self-identify as men.	Transcripts from the first 20 interviews inform the CHI note; the remaining interview transcripts will be analyzed in conjunction with the first 20 interviews to expand on the CHI note and write a journal article.
5-6 focus groups with all genders who attend unsponsored editing, meet-up, and/or mentoring events	3 small focus groups conducted at Wikimania London 2014	Transcripts from these focus groups will inform an upcoming journal article.
Online survey results	90 complete surveys	Although Qualtrics was easy to use to set up the survey, survey respondents found it difficult. It timed out. It crashed on certain browsers. Sadly, I received only 90 surveys that were 100% complete. However, I believe the data received and shared here can help inform new surveys.
Summary of existing efforts to address the gender gap for English language Wikipedia	Incomplete	My scope was too large, and my focus shifted. I was unable to complete this analysis.

Global Metrics

Because my project was focussed on research, the following metrics are more difficult to measure. However, this IEG did generate interest and discussion--as well as participation in--the Wikipedia communities.

Metric	Achieved outcome	Explanation
1. Number of active editors involved	~30	Editors were involved in interviews and feedback.
2. Number of new editors	8	The six students who helped with coding also created user accounts and made a few edits. Two new editors joined as a result of the I Love To You edit-a-thon. Please note: Recruiting new editors was not a focus of this IEG.
3. Number of individuals involved	~35	Editors involved in interviews and feedback; WMF staff involved in feedback, revisions of blog posts, interviews, etc.
4. Number of new images/media added to Wikimedia articles/pages	N/A	The goal of this IEG was to do research. It's difficult to estimate content added to Wikimedia as a result.
5. Number of articles added or improved on Wikimedia projects	N/A	The goal of this IEG was to do research. It's difficult to estimate content added to Wikimedia as a result.
6. Absolute value of bytes added to or deleted from Wikimedia projects	N/A	The goal of this IEG was to do research. It's difficult to estimate content added to Wikimedia as a result.

Learning question: Did your work increase the motivation of contributors, and how do you know?

Again, this wasn't the goal of the IEG. I've received some positive feedback from interview participants and from edit-a-thon attendees, but I have not done follow-up work regarding whether this IEG directly increased the motivation of contributors. For some, it certainly gave them an opportunity to be heard.

Indicators of impact

Do you see any indication that your project has had impact towards Wikimedia's strategic priorities? We've provided 3 options below for the strategic priorities that IEG projects are mostly likely to impact. Select one or more that you think are relevant and share any measures of success you have that point to this impact. You might also consider any other kinds of impact you had not anticipated when you planned this project.

This IEG contributed to understanding how participation might be increased and why some editors who self-identify as women struggle to remain in the English language Wikipedia communities in which they have been contributing.

Project resources

Interview Data

Due to the sensitive nature of the topic and in respect of the majority of my participants' wishes as well as in deference to my IRB, I will not release the interview transcripts. However, the research note below is based on the first 20 transcripts.

"The Heart Work of Wikipedia: Gendered, Emotional Labor in the World's Largest Online Encyclopedia" by Amanda Menking & Ingrid Erickson, CHI 2015

Survey Data

Raw survey data

Gendergap Mailing List Analysis Data

Images of Gendergap mailing list social network analysis
Gendergap mailing list codebook
Gendergap mailing list coded months
Python script to scrap mailing list for specific fields to perform social network analysis (created by Michael Gilbert)
Gendergap mailing list social network analysis spreadsheet listing participants and edges (per year)

Presentations and Posters

Learning

Snowball sampling and gaining access to the Wikipedia communities took time and was challenging in some respects, but it also worked well. I would highly recommend that researchers who are not Wikipedians spend time lurking on mailing lists, editing articles, participating in talk page conversations, and generally learning about different aspects of the movement through participation.

What worked well

Created

https://meta.wikimedia.org/wiki/Grants:Learning_patterns/Conducting_a_semi-structured_interview

Endorsed

What didn’t work

Content analysis

This was primarily a scope issue. I would love to do content analysis in the future to see how framing of the "gender gap" has evolved.

Interviews with "trolls"

I had wanted to interview users who had been identified or self-identified as "trolls," especially "trolls" who engaged in gender-based hostility. Unfortunately, I was unsuccessful in recruiting these individuals to speak with me. Partly this may have been a matter of scope (this was not within my initial plan) and funding (compensation for time may have motivated certain groups of interviewees).

Other recommendations

I would recommend that anyone doing research and/or community engagement in this area prioritize self-care. I read posts about my IEG on Wikipediocracy. I also read Facebook comments in response to my WMF blog posts. Despite overwhelming support from WMF, many Wikipedians, and the majority of interviewees, negative feedback and engaging in online conversations (whether on the Discussion pages here or elsewhere) was time-consuming and, at times, emotionally exhausting. This is a part of academic life in general, but it can be particularly draining in the context of Wikipedia.
Abiding by the new Open Access Policy requires time and planning. While my IEG was granted before the policy was instituted, I've tried to honor it and to make the majority of the data collected and analyzed during this IEG available for others. However, this has taken more time than I first estimated. I would recommend new IEG recipients set up Figshare, Git Hub, etc. accounts ahead of time and share data as they make progress.

Next steps and opportunities

Other community members and/or researchers may want to use the raw survey data to structure additional surveys.
Other community members and/or researchers may want to use the Gendergap mailing list codebook, coded data, and/or social network analysis data to continue to investigate questions re: the success of and different approaches to mailing lists as well as questions of participation.
I plan to continue to do interviews, to expand the CHI note into a journal article, and to continue to participate in discussions and research re: the "gender gap."

Think your project needs renewed funding for another 6 months?

Part 2: The Grant

Finances

Actual spending

Expense	Approved amount	Actual funds spent	Difference
Travel	$3500	$2720.61	$779.39
Transcriptions	$1200	$682.50	$517.50
Survey development	$375	$0	$375
Researcher stipend	$3000	$3000	$0.00
Student renumeration/stipend	$0.00	$826.75	($826.75)
Total	$8075	$7229.86	$845.14

Remaining funds

Do you have any unspent funds from the grant?

Please answer yes or no. If yes, list the amount you did not use and explain why.

Yes, I overestimate travel expenses, and I didn't have all of the interview audio professionally transcribed. Also, I used Qualtrics to develop a survey rather than hiring someone to do it.

If you have unspent funds, they must be returned to WMF. Please see the instructions for returning unspent funds and indicate here if this is still in progress, or if this is already completed:

Funds have been returned with interest: $845.20.

Documentation

Did you send documentation of all expenses paid with grant funds to grantsadminwikimedia.org, according to the guidelines here?

Please answer yes or no. If no, include an explanation.

Yes.

Confirmation of project status

Did you comply with the requirements specified by WMF in the grant agreement?

Please answer yes or no.

Yes.

Is your project completed?

Please answer yes or no.

In many ways, yes. In many other ways, no. The "gender gap" is a complex issue impacted by socio-technical norms, cultural influences, and individual differences. I intend to continue this research and share findings with the Wikipedia communities and WMF.

Grantee reflection

I learned a great deal during this IEG. In retrospect, I would have limited my scope. While the Wikipedia communities would benefit from a content analysis of all of the discussion and documentation of the "gender gap" (and attempts to address it), I was not able to do this work during my IEG. Also, since the discussion and documentation is currently in a state of flux, especially given the Inspire campaign, I think there may be a time in the future that is better suited to looking at historical snapshots.

Overall, I'm extremely pleased to see that the Wikipedia communities and WMF are investing more time, attention, and funding to consider the "gender gap." I began this research because of Adrianne Wadewitz's HASTAC blog post (quoted above). I'm indebted to her example, work, and words. I'm incredibly grateful for the support I've received from my PhD committee, including my advisor, David McDonald, and my co-author, Ingrid Erickson. I'm also grateful for the support I've received from many at WMF and in the Wikipedia communities.