Research:Newsletter/2021/June

Vol: 11 • Issue: 06 • June 2021 [contribute] [archives]

Feminist critique of Wikipedia's epistemology, Black Americans vastly underrepresented among editors, Wiki Workshop report: Summaries of 26 new research publications

By: Markworthen and Tilman Bayer

"WP:NOT, WP:NPOV, and Other Stories Wikipedia Tells Us: A Feminist Critique of Wikipedia’s Epistemology"

Reviewed by Markworthen

This paper by Menking and Rosenberg, published in the journal Science, Technology, & Human Values, is a recondite article. Its depth is both a strength (diligent study of the article will likely enhance Wikipedians' understanding of potential problems, such as our assumptions about what constitutes a reliable source or our epistemological presumptions), and a weakness (most Wikipedians will not read it because it is so dense).

I tried (several times) but cannot improve on the authors' summary. Here, then, is an excerpt from the article abstract:

A repository for established facts, Wikipedia is also a social space in which the facts themselves are decided. As a community, Wikipedia is guided by the five pillars—principles that inform and undergird the prevailing epistemic and social norms and practices for Wikipedia participation and contributions. We contend these pillars lend structural support to and help entrench Wikipedia’s gender gap as well as its lack of diversity in both participation and content. In upholding these pillars, Wikipedians may unknowingly undermine otherwise reasonable calls for inclusivity, subsequently reproducing systemic biases. We propose an alternative set of pillars developed through the lens of feminist epistemology ... Our aim is not only to reduce bias, but also to make Wikipedia a more robust, reliable, and transparent site for knowledge production.

Context

Background reading that will enhance understanding of this Menking & Rosenberg (2021) article:

• Benjakob, Omer and Stephen Harrison. From Anarchy to Wikiality, Glaring Bias to Good Cop: Press Coverage of Wikipedia's First Two Decades. The Signpost 17, no. 1 (31 January 2021).
• Epistemology
• Feminist epistemology
• Feminist method
• Lorraine Code and Helen Longino – Feminist philosophers whose ideas helped guide Menking & Rosenberg's analysis.

Talk page discussions about the article

The article has generated some engaging discussions on Wikipedia talk pages, for example:

• "Helping new editors: Learn Wikipedia's unique culture" – Discussion of the article begins after I posted the citation on 28 December 2020 @ 17:45 UTC (in that thread).
• "WP:5P sidetrack (part II) on Iridescent's Talk page.
• w:User talk:Markworthen/sandbox/Feminist critique of Wikipedia's epistemology

African Americans are vastly underrepresented among US Wikimedians, but contribute motivated by "black altruism"

Reviewed by Tilman Bayer

Last month, the Wikimedia Foundation published the results of its annual "Community Insights", a global survey of 2,500 Wikimedians (including active editors and program leaders) conducted in September/October 2020.

For the first time, the survey asked about race and ethnicity, confined to two countries where such categories are widely used and accepted: the US (195 responses) and the UK (67). Among US contributors, the findings shows striking gaps among Black/African American editors (0.5% compared to 13% among the general population) and American Indian/Alaska Native editors (0.1% vs 0.9%). Hispanic/Latino/a/x editors show a lesser but still large gap (5.2% vs. 18%). White/Caucasian (89% vs. 72%) and especially Asian Americans (8.8% vs. 5.7% among the general population) are over-represented among contributors in the US.

Survey findings about the race of contributors in the US, compared to the overall population (categories are overlapping and thus sum to more than 100%)

In the UK, the survey similarly found "significant underrepresentation" of Black or Black British editors (0.0% vs. 3.0% in the general population), whereas the percentage of white editors was close to the general population.

A racial or ethnic gap among Wikipedians in the US has long been anecdotally observed or conjectured (see e.g. this 2010 thread which also contained some informed speculation about possible reasons), but this marks the first time that it is backed by empirical survey data, related to the fact that the Wikimedia Foundation's annual surveys are global in nature and there are no internationally accepted definitions of race and ethnicity (or worse, survey questions of this nature would be considered offensive in many countries) [1][2].

Illustration from the 2018 "Pipeline of Online Participation Inequalities" paper

Correspondingly, there has been few research about possible reasons for such gaps. An exception is the 2018 paper "The Pipeline of Online Participation Inequalities: The Case of Wikipedia Editing", which we previously reviewed here with a more general focus, but it also contains some insights about reasons why African Americans contribute at a lower rate.

While this study did not find a significant racial disparity among the earlier parts of the pipeline (measuring whether survey respondents had heard of Wikipedia or had visited Wikipedia), when it comes to "know[ing] that Wikipedia can be edited [...] age, gender, and several racial/ethnic identity categories (Black, Hispanic, Other) emerge as salient explanatory factors where they did not before. Income no longer explains the outcome. Education level associates strongly with knowing Wikipedia can be edited." However, racial and ethnic background factors "do not associate with who contributes content" (i.e. the last part of the pipeline). This points to raising awareness of Wikipedia's editability as a potential strategy for reducing these gaps, although this would not address "the importance of education and Internet skills" gaps for closing knowledge gaps that the authors highlight in their overall conclusions.

Conversely, a 2020 paper answered the question "What drives Black contributions to Wikipedia?" with the following conclusions, based on a survey of 318 Black Wikipedia editors in the US:

First, black altruism [measured via the survey question "Writing/editing Wikipedia is a way for me to help the Black community," "Sharing my knowledge through Wikipedia improves content on the Black community," and "It is important that Blacks write/edit Wikipedia"] indirectly influences Wikipedians’ content contribution through their perception of information quality regarding Black content in the Wikipedia universe. In other words, the higher their altruistic tendencies, the stronger their perceptions of information quality. [...] We argue that black altruism mediated through perceptions of information quality [...] helps extend black digital culture in the online encyclopedia. Therefore, we suggest one of the gratifications of Black Wikipedia contribution is likely the production and dissemination of Black cultural information content. [...]

Secondly, our findings suggest that Black Wikipedians’ perceived social presence, is also a significant driver of content contribution. That is to say, Wikipedia contribution uplifts self-esteem and feelings of self-enhancement among Black authors and editors. [...]

Lastly our findings, demonstrated that entertainment is not a factor that significantly motivates Black Wikipedians’ content contribution in the current study. We suggest this may be because Wikipedia content contribution, like long-form blogging, requires “more sustained time, editing, maintenance” (Steele, 2018 p. 116), intellectual endeavor, and a higher degree of technological proficiency than other types of social media activities. Nevertheless, our results are a sharp contrast to previous studies which suggest that amusement is an influential factor in Wikipedia contributions [...]"

Survey respondents were recruited in 2017 via Qualtrics "based on predefined characteristics such as individuals who identified as Black/African American, resided in the United States, and had made at least one edit/contribution to Wikipedia's English edition over the last three years". Interestingly, the resulting sample of 318 Black Wikipedia contributors was much larger than that of the WMF Community Insights survey, which (barring some extreme downward adjustments during the weighting process) appears to have consisted of a single Black respondent in the sample, considering the stated percentage of 0.5% among 195 US-based respondents.

Wikiworkshop 2021

Report by Tilman Bayer

The annual WikiWorkshop, part of The Web Conference, took place as an online event on April 14, 2021, featuring the papers listed below. The organizers reported that 78% of attendees were non-native English speakers, 66% first-time attended Wiki Workshop for the first time, 53% were academic researchers and 34% students.

"References in Wikipedia: The Editors' Perspective"

From the abstract:

"we explore the creation and collection of references for new Wikipedia articles from an editor’s perspective. We map out the workflow of editors when creating a new article, emphasising on how they select references."

"Do I Trust this Stranger? Generalized Trust and the Governance of Online Communities"

From the abstract:

"we hypothesize that administrators’ community governance policy might be influenced by general trust attitudes acquired mostly out of the Wikipedia context. We use a decontextualized online experiment to elicit levels of trust in strangers in a sample of 58 English Wikipedia administrators. We show that low-trusting admins exercise their policing rights significantly more (e.g., block about 81% more users than high trusting types on average). We conclude that efficiency gains might be reaped from the further development of tools aimed at inferring users’ intentions from digital trace data."

"Negative Knowledge for Open-world Wikidata"

From the abstract:

"Like most major KBs [knowledge bases, Wikidata is] incomplete and therefore operates under the open-world assumption (OWA) – statements not contained in Wikidata should be assumed to have an unknown truth. The OWA ignores however, that a significant part of interesting knowledge is negative, which cannot be readily expressed in this data model. In this paper, we review the challenges arising from the OWA, as well as some specific attempts Wikidata has made to overcome them. We review a statistical inference method for negative statements, called peer-based inference, and present Wikinegata, a platform that implements this inference over Wikidata. ... Wikinegata is available at https://d5demos.mpi-inf.mpg.de/negation ."

"A Brief Analysis of Bengali Wikipedia's Journey to 100,000 Articles"

From the abstract:

"This paper analyzes the various associating factors throughout this journey including the number of active editors, number of content pages, pageview, etc., along with the connection to outreach activities with these parameters."

"WikiShark: An Online Tool for Analyzing Wikipedia Traffic and Trends"

From the abstract:

"This paper introduces WikiShark (www.wikishark.com) – an online tool that allows researchers to analyze Wikipedia traffic and trends quickly and effectively, by (1) instantly querying pageview traffic data; (2) comparing traffic across articles; (3) surfacing and analyzing trending topics; and (4) easily leveraging findings for use in their own research."

"Tracing the Factoids: the Anatomy of Information Re-organization in Wikipedia Articles"

From the abstract:

"... we investigate the impact of gradual edits on the re-positioning and organization of the factual information in Wikipedia articles [...] we show that in a Wikipedia article, the crowd is capable of placing the factual information to its correct position, eventually reducing the knowledge gaps. We also show that the majority of information re-arrangement occurs in the initial stages of the article development and gradually decreases in the later stages."

"Wikidata Logical Rules and Where to Find Them"

From the paper (an extended abstract):

"We are interested in soft (approximate) constraints expressed as dependencies (or logical rules), such as the constraint that “a person cannot be born after one of her children”. Such rules have proven to be useful for error detection [4], adding missing facts [3], executing queries faster, and reasoning [1]. Not only these rules are not stated in Wikidata, but, to the best of our understanding, a way to express them as constraints is still to be defined in the repository ... there are very few rules that are exact, i.e., true for each and every case. As an example, consider a rule stating that “a country has always one capital”. This is true for most countries, but there are 15 countries that have two or more capitals. Therefore, the rule has a very high confidence, but it is not exact ... The goal of our work is to create a large collection of rules for Wikidata with their confidence measure. In this abstract, we report on two directions we have been exploring to obtain such rules, our results, and how we believe the Wikimedia community could benefit from this effort."

"Simple Wikidata Analysis for Tracking and Improving Biographies in Catalan Wikipedia"

From the abstract:

"we highlight the possibilities of taking advantage of structured data from Wikidata for evaluating new biographical articles, so facilitating users to get engaged into diversity challenges or track potential vandalism and errors"

Related code: https://github.com/toniher/wikidata-pylisting

"Structural Analysis of Wikigraph to Investigate Quality Grades of Wikipedia Articles"

From the abstract:

"we present a novel approach based on the structural analysis of Wikigraph to automate the estimation of the quality of Wikipedia articles. We examine the network built using the complete set of English Wikipedia articles and identify the variation of network signatures of the articles with respect to their quality. Our study shows that these signatures are useful for estimating the quality grades of un-assessed articles with an accuracy surpassing the existing approaches in this direction."

"Towards Open-domain Vision and Language Understanding with Wikimedia"

From the abstract:

"This work [i.e. research proposal] describes a project towards achieving the next generation of models, that can deal with open-domain media, and learn visio-linguistic representations that reflect data’s context, by jointly reasoning over media, a domain knowledge-graph and temporal context. This ambition will be leveraged by a Wikimedia data framework, comprised by comprehensive and high-quality data, covering a wide range of social, cultural, political and other type of events"

"Language-agnostic Topic Classification for Wikipedia"

From the abstract:

"we propose a language-agnostic approach based on the links in an article for classifying articles into a taxonomy of topics that can be easily applied to (almost) any language and article on Wikipedia. We show that it matches the performance of a language-dependent approach while being simpler and having much greater coverage."

See also: online demo, data dumps, model details

"Fast Linking of Mathematical Wikidata Entities in Wikipedia Articles Using Annotation Recommendation"

From the abstract:

"We evaluate the quality and time-savings of AI-generated formula and identifier annotation recommendations on a test selection of Wikipedia articles from the physics domain. Moreover, we evaluate the community acceptance of Wikipedia formula entity links and Wikidata item creation and population to ground the formula semantics. Our evaluation shows that the AI guidance was able to significantly speed up the annotation process by a factor of 1.4 for formulae and 2.4 for identifiers. Our contributions were accepted in 88% of the edited Wikipedia articles and 67% of the Wikidata items. The >>AnnoMathTeX<< annotation recommender system is hosted by Wikimedia at annomathtex.wmflabs.org. In the future, our data refinement pipeline will be integrated seamlessly into the Wikimedia user interfaces."

"ShExStatements: Simplifying Shape Expressions for Wikidata"

From the abstract:

"Wikidata recently supported entity schemas based on shape expressions (ShEx). They play an important role in the validation of items belonging to a multitude of domains on Wikidata. [...] In this article, ShExStatements is presented with the goal of simplifying writing the shape expressions for Wikidata."

"Inferring Sociodemographic Attributes of Wikipedia Editors: State-of-the-art and Implications for Editor Privacy"

From the abstract:

"we investigate the state-of-the-art of machine learning models to infer sociodemographic attributes of Wikipedia editors based on their public profile pages and corresponding implications for editor privacy. [...] In comparative evaluations of different machine learning models, we show that the highest prediction accuracy can be obtained for the attribute gender, with precision values of 82% to 91% for women and men respectively, as well as an averaged F1-score of 0.78. For other attributes like age group, education, and religion, the utilized classifiers exhibit F1-scores in the range of 0.32 to 0.74, depending on the model class."

"The Language of Liberty: A preliminary study"

From the paper:

"We managed to align more than 37,000 articles across Wikipedia and Conservapedia; of these, about 28,000 pages share an identical title, while the remaining ones are aligned based on redirect pages. In total, the whole corpus contains 106 million tokens and 558,000 unique words. [...] We can notice marked differences in word usage in the two resources: Wikipedia authors tend to use more objective/neutral words (affordable care, american politician), in addition to many non-political terms. In Conservapedia [there] prevail derogatory terms such as rino, which stands for “Republican In Name Only", and Democrat Party, but also topics of high concern to the conservative community such as the homosexual agenda, communist manifesto, and fetal tissue."

"Information flow on COVID-19 over Wikipedia: A case study of 11 languages"

From the abstract:

"we study the content editor and viewer patterns on the COVID-19 related documents on Wikipedia using a near-complete dataset gathered of 11 languages over 238 days in 2020. Based on the analysis of the daily access and edit logs on the identified Wikipedia pages, we discuss how the regional and cultural closeness factors affect information demand and supply."

"Towards Ongoing Detection of Linguistic Bias on Wikipedia"

From the abstract:

"As part of our research vision to develop resilient bias detection models that can self-adapt over time, we present in this paper our initial investigation of the potential of a cross-domain transfer learning approach to improve Wikipedia bias detection. The ultimate goal is to future-proof Wikipedia in the face of dynamic, evolving kinds of linguistic bias and adversarial manipulations intended to evade NPOV issues."

Analysis of two million AfD (Article for Deletions) discussions

From the abstract:

"[Wikipedia article deletion] decisions (which are known as “Article for Deletion”, or AfD) are taken by groups of editors in a deliberative fashion, and are known for displaying a number of common biases associated to group decision making. Here, we present an analysis of 1,967,768 AfD discussions between 2005 and 2018. We perform a signed network analysis to capture the dynamics of agreement and disagreement among editors. We measure the preference of each editor for voting toward either inclusion or deletion. We further describe the evolution of individual editors and their voting preferences over time, finding four major opinion groups. Finally, we develop a predictive model of discussion outcomes based on latent factors."

Among the findings are that "Editors who joined before 2007 tend to overwhelmingly belong to the more central parts of the network" and that "user preferences [for keep or delete] are relatively stable over time for ... more central editors. However, despite the overall stability of trajectories, we also observe a substantial narrowing of opinions in the early period of an AfD reviewer tenure. ... Strong deletionists exhibit the least amount of change, suggesting the possibility of lower susceptibility, or higher resistance, to opinion change in this group." Overall though, the authors conclude that "differences between inclusionists and deletionists are more nuanced than previously thought."

"Assessing the quality of health-related Wikipedia articles with generic and specific metrics"

From the abstract:

" we use general and health-specific features from Wikipedia articles to propose health-specific metrics. We evaluate these metrics using a set of Wikipedia articles previously assessed by WikiProject Medicine. We conclude that it is possible to combine generic and specific metrics to determine health-related content’s information quality. These metrics are computed automatically and can be used by curators to identify quality issues."

"Wikipedia Editor Drop-Off: A Framework to Characterize Editors' Inactivity"

A figure from the paper, showing "The different states of drop-off related to activity and their possible transitions"

From the abstract:

"... we present an approach to characterize Wikipedia’s editor drop-off as the transitional states from activity to inactivity. Our approach is based on the data that can be collected or inferred about editors’ activity within the project, namely their contributions to encyclopedic articles, discussions with other editors, and overall participation. Along with the characterization, we want to advance three main hypotheses, derived from the state of the art in the literature and the documentation produced by the community, to understand which interaction patterns may anticipate editors leaving Wikipedia: 1) abrupt interactions or conflict with other editors, 2) excess in the number and spread of interactions, and 3) a lack of interactions with editors with similar characteristics."

The paper is part of an ongoing research project funded by a €83,400 project grant from the Wikimedia Foundation. Some related code can be found at https://github.com/WikiCommunityHealth/ .

Wikimedia Foundation Research Award of the Year

Besides presentations about the papers listed above, the Wiki Workshop event also saw the announcement of the first "Wikimedia Foundation Research Award of the Year" ("WMF-RAY", cf. call for nominations), with the following two awardees:

"Content Growth and Attention Contagion in Information Networks: Addressing Information Poverty on Wikipedia" (also presented at last year's Wikiworkshop), a paper which according to the laudators

"demonstrates causal evidence of the relationship between increases in content quality in English Wikipedia articles and subsequent increases in attention. The researchers conduct a natural experiment using edits done on English Wikipedia via the Wiki Education Foundation program. The paper shows that English Wikipedia articles that were improved by students in the program gained more viewers than a group of otherwise similar articles. It also found that this effect spills over into a range of articles linked to from the improved articles."

"Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages" and Masakhane (which describes itself as "A grassroots NLP community for Africa, by Africans"). The paper

describes a novel approach for participatory research around machine translation for African languages. The authors show how this approach can overcome the challenges these languages face to join the Web and some of the technologies other languages benefit from today."

While the research does not seem to have concerned Wikipedia directly, the laudators find it an "inspiring example of work towards Knowledge Equity, one of the two main pillars of the 2030 Wikimedia Movement Strategy" and expect the project's success

"will directly support a range of Wikimedia Foundation and Wikimedia Movement goals including the newly-announced Abstract Wikipedia which will rely heavily on machine translation too."

Consistent with its title, the paper features an impressive list of no less than 48 authors (with the cited eprint having been submitted to arXiv by Julia Kreutzer of Google Research).

Briefly

• See the page of the monthly Wikimedia Research Showcase for videos and slides of past presentations.

References

Supplementary references and notes:

Wikimedia Research Newsletter
Vol: 11 • Issue: 06 • June 2021
About • Subscribe: Email • [archives] • [Signpost edition] • [contribute] • [research index]