Research:Newsletter/2016/February

Vol: 6 • Issue: 02 • February 2016 [contribute] [archives]

Wikipedia and paid labour; Swedish gender gap; how verifiable is "verifiable"?

With contributions by: Nicolas Jullien, Piotr Konieczny, and Tilman Bayer

"Monetary materialities of peer-produced knowledge: the case of Wikipedia and Its Tensions with Paid Labour"

Reviewed by Nicolas Jullien

This article discusses the links between paid efforts and voluntary efforts in the development of Wikipedia, focusing on the question of paid editing. It stresses the fact that Wikipedia is a mixed economy that results partly from paid labor (the technostructure and the people in charge of maintaining it, and those who defend the project in court, i.e. the paid employees of the WF).

The core of the article discusses, based on the debate about Wiki-PR (a company which was paid by firms to "edit" their EN-Wikipedia pages), and the impact it had on Wikipedia policy. It sheds light on the discussion between the Foundation, which expressed a more strict interpretation of the rules, and the contributors, especially from non-English Wikipedias, that took a more "pragmatic" approach. Paid editors provided help to the smaller projects in terms of creation of knowledge. The analysis, which views Wikipedia as a sort of communist organization, is less convincing, as is the fact that the authors did not compare this debate with what happens in FLOSS (free-libre open-source software) or in the non-digital world (the Foundation, or the local community groups), which are other example of the co-existence of voluntary and paid work.

"The Swedish Wikipedia gender gap"

Reviewed by Piotr Konieczny

This masters thesis focuses on the Swedish Wikipedia and its gender gap. It quantifies data and provides information about why Swedish women are not contributing to the project. The author collected data through a questionnaire advertised in December 2014 on the Swedish Wikipedia through a project-wide banner (promotion that an average researcher can only dream about when it comes to English Wikipedia). The paper estimates the Swedish Wikipedia gender gap in the form of the percentage of female editors at between 13% to 19%, based on the self-reported data from Wikipedia account profiles and answers to the questionnaire. More interesting is the analysis of the activity of the accounts: the self-declared male accounts are several times more active then the female accounts, with the authors estimating that only about 5% of the site's content is written by women. Contrary to some prior research (most of which focused on the English Wikipedia), the Swedish Wikipedia's editors and readers do not perceive Wikipedia as a place where sexist comments are significant, though about a third agree that general conflicts between editors do take place. Nonetheless, women are less likely than men to think (1) that Wikipedia is welcoming to beginners; (2) that everyone gets treated equally, regardless of gender; (3) that editing means taking on conflicts. Women are more likely than men to acknowledge the existence of sexist comments. In the author's own words, "women have more concerns about the community being sexist and not welcoming, and do not expect conflict as part of editing to the same degree as men", though the author also notes that statistical tests suggest that "the differences in opinion between gender groups do not differ [sic] greatly".

The author concludes that there is no evidence that the Swedish Wikipedia's readers have any preconceived negative notions about the Wikipedia community (such as "it is sexist") that should inhibit potential women contributors from editing and thus contribute to the gender gap. He states: "Significant differences in perceived competence were found. Women report 'I’m not competent enough' as a strong contributing factor to them not editing more than twice as often as men." The author suggests that because women often perceive, whether correctly or not, that they have lower computer skills than men, and see Wikipedia as a website which requires above-average computer skills, this (rather than an unfriendly, sexist community) may be the most significant factor affecting their lack of contributions. (Cf. related coverage: "Mind the skills gap: the role of Internet know-how and gender in differentiated contributions to Wikipedia'", "Does advertising the gender gap help or hurt Wikipedia?")

Test of 300k citations: how verifiable is "verifiable" in practice?

Reviewed by Tilman Bayer

Four researchers from Dartmouth College have taken the requirement of "verifiability", one of Wikipedia's core content policies, literally. Their preprint examines 295,800 citations from the 5000 most viewed articles on the English Wikipedia (out of a larger set of 23 million citations extracted from a July 2014 dump). These comprised both inline citations (footnotes) and "free citations" (those not related to any particular part of the article). The authors conclude that

"while the quality of references in the overall sample is reasonably high, verifiability varies significantly by article, particularly when emphasizing the use of standard digital identifiers and taking into account the practical availability of referenced sources."

Unsurprisingly, the study did not examine whether the cited documents actually match the information in the articles. Rather, it concerns the question whether the citation enables the reader to carry out this verification. The authors argue that

"simply providing citations and references does not automatically guarantee verifiability. Whether or not provided references and citations are accessible ... is just as important as providing the reference or citation in the first place. There are many ways that an online information source might provide citations and references and still be difficult to verify."

They divide these difficulties into two categories: "technical verifiability" and "practical verifiability."

Technical verifiability is defined as "the extent to which a reference provides supporting information that permits automated technical validation of the existence of the referenced material, based on existing technical standards or conventions," concretely ISBNs, DOIs and Google Books IDs. The study found that:

• "Out of 37,269 book citations, 29,736 book citations (79.8%) had valid ISBNs, while 3,145 (8.4%) of book citations had invalid ISBNs, and 4,388 book citations (11.8%) contained no ISBN information."
• "Out of 14,081 Google Books-containing citations, 3,159 (22.4%) contained invalid Google Books IDs."
• "presence or absence of a Digital Object Identifier (DOI) was noted for any reference tagged as‘journal’, ‘study’, ‘dissertation’, ‘paper’, ‘document’, or similar. Out of 41,244 of these citations, only 5,337 (12.9%) contained neither a DOI or a link to a known open access journal."

Practical verifiability is defined as "the extent to which referenced material is accessible to someone encountering the reference." In particular, the authors point out that information supported by a paywalled journal article "is practically unverifiable to someone without the additional means to access the supporting journal article. Similarly, if an ISBN is present but refers to a book that only has one extant copy in a library thousands of miles away, then the information it supports is practically unverifiable to someone without the additional means to access the supporting book." Apparently the authors found it difficult to translate these notions into criteria that would lend themselves to a large scale quantitative analysis, and settled for two rather narrowly defined but still interesting aspects:

• "Journal citations linking to ‘arXiv' and 'PubMed Central (PMC)' were taken to be open access, while all others were marked unconfirmed. 5,275 of the journal citations out of 41,244 (12.8%) belonged to this confirmed open access category, while 30,632 (74.3%) contained some digital identifier but were not confirmed to be open."
• "Out of the 10,922 working Google Books links, most (7,749, or 71.0%) are partially viewable with samples, while 1,359 (12.4%) are fully viewable and 1,814 (16.6%) are not viewable at all."

The preprint also contains a literature overview about information quality on Wikipedia, which does the topic scant justice (e.g. of the only three mentioned systematic studies of article accuracy, one is the well-known but over a decade old Nature study, another is a 2014 article whose methodology and conclusions have been described as very questionable, see also below).

With some caveats, e.g. that the quality of the 5000 most-viewed English Wikipedia articles might differ from the quality of the average article, the authors conclude that "from the perspective of overall quality of references in Wikipedia, these findings might seem encouraging", but are concerned that many citations are not practically verifiable.

Twelve years of Wikipedia research

Reviewed by Tilman Bayer

This short (two-page) paper presents "preliminary results that characterize the research done on and using Wikipedia since 2002". It is based on a dataset of 3582 results of a Scopus search in November 2013 (for the term "Wikipedia" in title, abstract and keywords), largely relying on the abstracts of these publications. 641 of them were discarded as unrelated. Of the remaining 2968, the relevance for Wikipedia was judged as "major" for 2301 and as "minor" for 667.

Examining a dichotomy that is familiar to the editors of this newsletter too (which, for example, usually does not cover papers that merely rely on Wikipedia as a text corpus, even though these are numerous in fields such as computer linguistics), the authors write:

"In terms of topic, there were almost an equal number of items about Wikipedia (1431, 48%) as there were using Wikipedia (1537, 52%)",

defining the latter as employing "Wikipedia either as a source/resource for other research or used Wikipedia to test the feasibility and applicability of tools or methods developed for purposes not directly related to Wikipedia". Those papers only began appearing in 2005, but overtook the "about" category in 2009 and have remained in the majority since." (See also coverage of a presentation at Wikimania 2013 that likewise traced publication numbers over the years – based on Google Scholar instead of Scopus – and dated the first appearance of "Wikipedia as a corpus" research to 2005, too: "Keynote on applicable Wikipedia research")

The researchers classified publications by their methodology, into "social/theoretical" (including "analyses and visualizations of Wikipedia") and "technological" (in the "about" category, this classification was reserved to "tools developed for improving Wikipedia"), and found that:

"the technological approach was considerably more popular (1856 items, 63%) compared to the social approach (1112 items, 37%). ... we see that at first the social aspects were emphasized, but since 2007 papers on technological aspects are much more frequent."

The authors extended their search beyond Scopus to Web of Science and the ACM Digital Library for an examination of how the overall volume of published Wikipedia research has developed over time. The resulting chart indicates that the fast growth of earlier years leveled off, with even some decrease in 2013, the last year examined.

Further criticism of study that had criticized accuracy of medical Wikipedia articles

Reviewed by Tilman Bayer

Three letters to the editor of the Journal of the American Osteopathic Association adds to criticism of an article by Hasty et al. that had appeared in the same journal earlier, and was widely covered in the media with headline phrases such as "90% of [Wikipedia's] medical entries are inaccurate".

Like editors from WikiProject Medicine at the time, the writers of the first letter lament that the paper's authors "have not made their dataset public, so it is impossible to confirm the veracity of their conclusions"; however, "they did share with us a small subset of their dataset on major depressive disorder. We closely examined two statements from Wikipedia that the researchers identified as inaccurate." After outlining that the peer-reviewed literature on these two issues is "rife with debate", and pointing out that some of it supports rather than contradicts the information on Wikipedia, they state that "It seems problematic to conclude that statements made in Wikipedia are wrong based on peer-reviewed literature", also quoting the editors of Nature observing that "peer review per se provides only a minimal assurance of quality". (On another occasion, the lead author had revealed a third Wikipedia statement that according to the study contradicted the peer-reviewed literature and which he described as dangerously wrong; however, it was in agreement with the hypertension guidelines of the UK National Institute for Health and Care Excellence (NICE).)

The letter writers highlight the fact that the study relied on "third-year residents with no specific expertise [to] correctly ascertain the accuracy of claims made on Wikipedia" in this way. In a response, Hasty et al. acknowledged that the peer-reviewed literature contained diverging viewpoints on the topic, but held that "if Wikipedia articles are considered review articles, then it would be expected that major controversial points would be discussed rather than presented from one perspective."

The second letter criticizes that "Because Hasty et al did not identify a specified number of assertions for each condition and did not measure whether Wikipedia and peer-reviewed literature were correct or not, respectively, their use of the McNemar test to compare Wikipedia vs peer-reviewed medical literature was inappropriate." A third letter also criticized the usage of this statistical test, adding that "I believe that the study here was incorrectly analyzed and inappropriately published through the same peer-review process that Hasty et al are holding to such high esteem. " In their response Hasty et al. defended their method, while acknowledging that "for greater clarity" some tables should have been labeled differently.

With such severe criticism from several independent sources, it is hard not to see this 2014 paper by Hasty et al. as discredited. Unfortunately, it continues to be occasionally cited in the literature (as mentioned in the review of the "verifiability" paper above) and in the media.

Briefly

The attention economy of Wikipedia articles on news topics

Reviewed by Tilman Bayer

Comparison of topic attention (red and gold lines: average and median pageview numbers to neighboring pages, black line: traffic to the page itself) and creation of new pages linked to the topic (vertical black segments) for an expected event (2012 Summer Olympics, top) and an unexpected event (Hurricane Sandy, bottom). In the graphs on the right, "white nodes represent the neighbor articles predating 2012; colored nodes correspond to neighbors created in 2012. The size of the nodes is proportional to their yearly traffic volume; ... New articles tend to be peripheral to these networks."

A paper in Scientific Reports examined how the public attention to a news topic relates to the pageviews of the Wikipedia article about that topic, and the creation dates of related articles. As proxy for the general attention to the topic, the authors use traffic to pages "neighboring" the main article about the topic itself (i.e. linking to and linked from it), including the time before it was created. From the (CC BY licensed) paper:

"Our analysis is focused on the year 2012. We collected the neighbors of 93,491 pages created during that year. ... Which kinds of articles precede or follow demand for information? In Table 1 we list a few articles with the largest positive and negative bursts. Topics that precede demand (ΔV/V > 0) tend to be about current and possibly unexpected events, such as a military operation in the Middle East and the killing of the US ambassador to Libya. These articles are created almost instantaneously with the event, to meet the subsequent demand. Articles that follow demand (ΔV/V < 0) tend to be created in the context of topics that already attract significant attention, such as elections, sport competitions, and anniversaries. For example, the page about Titanic survivor Rhoda Abbott was created in the wake of the 100th anniversary of the sinking."

A Swiss perspective on Wikipedia and academia

Reviewed by Piotr Konieczny

This conference paper states in its abstract an intent to broadly analyze and present all aspects of Wikipedia use in education. Unfortunately, it fails to do so. For the first four and half pages, the paper explains what Wikipedia is, with next to no discussion of the extensive literature on the use of Wikipedia in education or its perceptions in academia. There is a single paragraph of original research, based on the interview of three Swiss Wikipedians; there is little explanation of why those people where interviewed, nor are there any findings beyond description of their brief editing history. The paper ends with some general conclusions. Given the semi-formal style of the paper, this reviewer finds that it resembles an undergraduate student paper of some kind, and it unfortunately adds nothing substantial to the existing literature on Wikipedia, education and academia.

Other recent publications

A list of other recent publications that could not be covered in time for this issue – contributions are always welcome for reviewing or summarizing newly published research.

• "The success and failure of quality improvement projects in peer production communities" From the abstract: "Mining data from five quality improvement projects in the English Wikipedia [ Collaboration of the Week (CotW), WikiCup, Wikipedia Education Program (WEP), Wikipedia:Community Portal (CP) and Today's Article for Improvement (TAFI) ] we show that certain types of strategies (e.g. creating artefacts from scratch) have better quality outcomes than others (e.g. improving existing artefacts), even if both are done by a similar cohort of participants."
• "Passing on: reader-sourcing gender diversity in Wikipedia" From the abstract: "We present the Passing On system, that reader-sources the creation and expansion of Wikipedia articles about women [using a database of New York Times obituaries], aiming to support frame changes on women's representation and offer a counter-public for novice Wikipedians." (See also http://passingon.natematias.com/)

From Joseph Priestley's A Chart of Biography (1765), referenced in this paper about biography networks on Wikidata

• "Quantifying cultural histories via person networks in Wikipedia" (also presented as conference poster at NetSci 2015) From the abstract: "At least since Priestley's 1765 Chart of Biography, large numbers of individual person records have been used to illustrate aggregate patterns of cultural history. Wikidata, the structured database sister of Wikipedia, currently contains about 2.7 million explicit person records, across all language versions of the encyclopedia. ... This situation provides us with the chance to go beyond the illustration of an idiosyncratic subset of individuals, as in the case of Priestly [sic]. ... We construct networks of co-occurring nationalities and occupations, provide insights into their respective community structure, and apply the results to select and color chronologically structured subsets of a large network of individuals, connected by Wikipedia hyperlinks." (See also coverage of earlier related work by the same authors: "The history of art mapped using Wikipedia")
• "Wikiometrics: a Wikipedia-based ranking system" From the abstract: "We demonstrate an innovative mining methodology, where different elements of Wikipedia – content, structure, editorial actions and reader reviews – are used to rank items in a manner which is by no means inferior to rankings produced by experts or other methods. We test our proposed method by applying it to two real-world ranking problems: top world universities and academic journals." (Cf. coverage of related papers coauthored by these authors: "'Do Famous People Live Longer?' Yes for academics, no for artists and athletes")
• "Enabling complex Wikipedia queries – technical report" From the abstract: "... we present a database schema used to store Wikipedia so it can be easily used in query-intensive applications. In addition to storing the information in a way that makes it highly accessible, our schema enables users to easily formulate complex queries using information such as the anchor-text of links and their location in the page, the titles and number of redirect pages for each page and the paragraph structure of entity pages." (Coauthored by one of the authors of the above mentioned ranking paper)
• "Wisdom of the crowd: Wikipedia controversies and coordinating policies" From the abstract: "Focusing on the years 2003–2006 of Wikipedia, this article discusses Wikipedia’s institutionalization process, which involved policy-setting with respect to two factors: the coordination of volunteer editors and external controversies."
• "Wikipedia and history: a worthwhile partnership in the digital era?" From the abstract: "... this paper examines Wikipedia as a mode of historical expression in the context of a project on the history of the Australian Paralympic Movement. Wikipedia’s key core content policies of verification, no original research, and neutral point of view (NPOV) as well as the collaborative premise that underpins the online encyclopaedia are the focal points of analysis. [...] the history of the Australian Paralympic Movement shows that Wikipedia can be important to history-making in the digital age in at least two ways. Wikipedia provides a mode of historical expression that is complementary to the narratives of traditional books, and the online encyclopaedia generates a community which has produced articles that have enhanced knowledge about the history of disability sport."
• "Translating the Swedish Wikipedia into Danish" From the abstract: " This paper presents a Swedish-Danish automatic translation system for Wikipedia articles (WikiTrans). Translated articles are indexed for both title and content, and integrated with original Danish articles where they exist. Changed or added articles in the Swedish Wikipedia are monitored and added on a daily basis. The translation approach uses a grammar-based machine translation system with a deep source-language structural analysis." (see also http://wikitrans.net/ )
• "Drawing questions from Wikidata" From the abstract: "We introduce Wikidata Quiz, an application that accesses the structured data set of knowledge base Wikidata. We construct a graph by querying multiple Wikidata items originating from any chosen topic."
• "Exploiting Wikipedia for information retrieval tasks" From the abstract: "This tutorial aims to provide a holistic view of Wikipedia's different features – text, links, categories, page views, editing history etc. – and explore the different ways they can be utilized in a machine learning framework. By presenting and contrasting the latest works that utilize Wikipedia in multiple domains, this tutorial aims to increase the awareness among researchers and practitioners in these fields to the benefits of utilizing Wikipedia in their respective domains ..."
• "Relation between Wikipedia edits and news published" From the abstract: "This research looks at the relation between the number of Wikipedia edits on corporate pages and the number of news published by English newspapers over a specified time span. ... The new insights could help companies generate and keep a good corporate image that helps with sales, customer loyalty or customer acquisition. Through showing that some scandals did affect the corporate Wikipedia pages, it can be stated that the site can act as a source of information for users or news outlets and that companies need to take Wikipedia as a public relations tool into account."
• "Getting a 'quick fix': first-year college students' use of Wikipedia" From the abstract: "This study found that first-year students are uncertain about the variety of ways to use information sources like Wikipedia, and that a direct and balanced approach to this area from instructors may lead to better outcomes than strict prohibition or silence."

References

Supplementary references:

Wikimedia Research Newsletter
Vol: 6 • Issue: 02 • February 2016
About • Subscribe: Email • [archives] • [Signpost edition] • [contribute] • [research index]