Research:Accuracy and quality of Wikipedia entries

Contact

Naomi Norman

EPIC

Imogen Casebourne

EPIC

Chris Davies

Oxford Universitiy

Michelle Fernandes

Oxford University

Research:Projects

This page documents a completed research project.

Key Personnel

This research is funded by a grant from the Wikimedia Foundation and carried out by a team of researchers based at Epic, a UK-based e-learning company, and at the University of Oxford. Key personnel on the project include:

Naomi Norman, Epic (PI)
Chris Davies, University of Oxford (co-PI)
Michelle Caroline Fernandes, University of Oxford (RA)
Imogen Casebourne, Epic

Project Summary

The objective of this project is to assess the accuracy and quality of a small sample of Wikipedia entries across a limited range of subjects and languages. The goals are:

to identify errors, omissions and other quality issues in a sample of Wikipedia articles
to compare the number of errors, omissions and other quality issues in a corresponding sample of articles on the same subjects from popular online encyclopedias for each language considered
to compare article quality between the two samples using both expert reviews and reader feedback.

The intention is that this pilot project will establish the best possible research approach to measure and surface the accuracy and quality of articles in Wikipedia, and to provide preliminary methodological recommendations towards a more comprehensive study of the quality of Wikipedia articles across subjects and languages.

Background Information

In December 2005, the scientific journal Nature, undertook a study to compare the accuracy of science entries on Wikipedia with those of the online version of Encyclopedia Britannica.^[1] The key difference between the two encyclopedias is that while Wikipedia relies on voluntary contributors, regardless of academic background, whereas entries in the Encyclopedia Britannica are compiled by selected paid expert advisors and editors. In forty-two reviews that were submitted, on average, four inaccuracies were identified per Wikipedia article in comparison to three per Encyclopedia Britannica article. Encyclopedia Britannica refuted the study’s findings, stating lapses in the methodology and asking for Nature to issue a public retraction of the article.^[2] Nature responded by rejecting Encyclopedia Britannica’s criticisms and refused to retract.^[3]

In the period following 2005, Wikipedia has been working hard with its readers and editors to improve the accuracy and quality of its entries. The current project is a small-scale preliminary study, inspired by Nature’s study, but that aims to employ greater rigor, involving academics and scholars, and examining more than just English language entries and in a broader range of subjects.

Methods

Recruitment

Graduate students from all academic disciplines at the University of Oxford will be contacted to participate in the study. The students will be asked to identify five or six respected academics in their field of study.

Sampling

Eight graduate students and twenty-four expert academics will comprise the review panel. The review panel will consist of native English, Spanish and Arabic speakers. The panel will be evenly distributed across four disciplines: humanities; social sciences; mathematical, physical and life sciences and medical sciences.

Review Process

Each student and expert academic will be asked to review two pairs of articles in their relevant subject domain and native language. In each pair, one article will be a Wikipedia entry and the other will an article from a popular alternative online encyclopedia. The students and expert academics will not be aware of the source of the articles. The articles will be assessed for a number of factors relating to the quality of the article. These will include comprehensiveness, accuracy, verifiability by references and objectivity. Articles will be rated using a survey tool specially designed for the study. The participants will be asked to justify their judgments, citing relevant references, and suggest appropriate corrections.

Results

The complete report is available under a CC-BY-SA 3.0 unported license.
The complete dataset produced by this study is also available under a CC0 license.

Dissemination

The research will be presented at relevant conferences and seminars (such as Wikimania, the Oxford University Department of Education). The output of this research will be published in an open access journal.

Wikimedia Policies, Ethics, and Human Subjects Protection

Our foremost priority is to conduct our research in an ethical, respectful, and non-disruptive manner. We will ensure that we conform to strict standards of informed consent and transparency in data collection methods. All participants will be informed about our affiliation, purpose and research goals. We will make all efforts to address any risks associated with participation in this study.

Benefits for the Wikimedia community

This work will help to:

identify errors, omissions and other quality issues in Wikipedia articles
determine comparisons in quality between Wikipedia entries and analogous entries from a popular alternative online version of an encyclopedia
compare reviews from readers with those of experts.

The results of this study will inform the design of a large-scale, empirical project across a range of languages and subject domains.

Time Line

November 2011

Sample and recruit students and academic experts
Identify pairs of articles of relevance

December 2011

Write and build article review

January 2012

Complete data collection
Collate relevant data from Wikipedia

February 2012

Code data
Data analysis

March 2012

Draft report
Submit final report

April 2012

Presentation of research

Funding

This project is supported by a grant from the Wikimedia Foundation.

References

↑ Giles, J. (2005) Internet encyclopaedias go head to head, Nature, vol.438, 15 December 2005, pp. 900-901.
↑ Encyclopædia Britannica, Inc. (March 2006), Fatally flawed: refuting the recent study on encyclopaedic accuracy by the journal Nature, [Online], Available at: http://corporate.britannica.com/britannica_nature_response.pdf [Accessed 11/03/11]
↑ Nature (23 March 2006), Encyclopaedia Britannica and Nature: a response, [Online], Available at http://www.nature.com/press_releases/Britannica_response.pdf [Accessed 11/03/11].

External links

Contacts

[1] Giles, J. (2005) Internet encyclopaedias go head to head, Nature, vol.438, 15 December 2005, pp. 900-901.

[2] Encyclopædia Britannica, Inc. (March 2006), Fatally flawed: refuting the recent study on encyclopaedic accuracy by the journal Nature, [Online], Available at: http://corporate.britannica.com/britannica_nature_response.pdf [Accessed 11/03/11]

[3] Nature (23 March 2006), Encyclopaedia Britannica and Nature: a response, [Online], Available at http://www.nature.com/press_releases/Britannica_response.pdf [Accessed 11/03/11].

[1]

[2]

[3]