Research:Wikipedia Primary School SSAJRP programme/Evaluation

Wikipedia Primary School SSAJRP programme

Timelines & progress

---

Method

Overview

Project evaluation main goals

• Analyzing the state of the art of topics related to Wikipedia Primary School Project
• Verifying the project impact in Wikipedia content
• Evaluating significant variables of interest for other research projects

Evaluation steps

1. Current state, Hypothesis, 2014
2. Intermediate state, Comparison, 2015
3. Final state, Conclusions, 2016

Hypothesis and questions:

• If an article is impossible to find it is useless. How easy is it to find an article?
  • Single article
    • Number of templates and categories
    • Number of visits (per categories and portals)
    • Featured in homepage as an article and as a fact
    • Links to the article and interlinks Portals and Wikiprojects Google Rank
  • Correlation with other articles
    • Total amount of articles in the same categories

• How to “calculate” the quality of an article through quantitative data? Is it possible to define variables in order to extract significant qualitative information?
  • Positive variables: Number of references and interlinks, Mentions, Portals and Wikiprojects, ecc.
  • Negative variables; Issues Stub, ecc.
  • Article: References and external links, Issues Templates, Edits
  • Direct evaluations: Features in homepage, Article quality, Monitoring, Mentions
  • Indirect evaluations: Links to the article, Number of visits, Page rank
  • Assigning points according to direct and indirect evaluations in order to generate a quality rank.

• Usually people write Wikipedia articles according to their interests. Can we drive interest?
  • Articles size (and variation)
  • Editor task force and casual contributors
  • Talk page
  • New users

Process

1. Define the list of 100 articles to evaluate
2. Extract data to visualize
3. Concept of the data visualization
4. First draft
5. Review and second iteration
6. Final data visualization

Preliminary analysis

Concept

The preliminary analysis aims to give a general view on the current status of the selected articles. The articles under analysis are 171.
The main goals of the intermediate analysis are the following:

• Understand the current state of the selected articles
• Define and visualize a set of parameters that can lead to the understanding of the ‘quality’ of the selected articles
• Explore the relations among articles

Tools

In order to do the visual evaluation, a scraper has been implemented. It’s name is Wikimole and it is available on GitHub for further implementation.
The following is the process Wikimole uses:

1. Wikimole gets data by using Wikipedia Api, a jQuery page scraper and some other external data sources such as Google Page rank and the Wikipedia article traffic statistics.
2. After an external section of data filtering and cleaning, Wikimole visualizes data by using Gephi and D3.js.

Intermediate analysis

Concept

The intermediate analysis provides an overview of the state of the selected articles at six months from the previous one.
The articles under analysis are now 176. Among them 36 have been reviewed thanks to the involvement of the Wikipedia Community, 25 articles have been reviewed by expert reviewers.

The main goals of the intermediate analysis are the following:

• Understand the impact of Wikipedia Primary School Project in the selected articles
• Provide actionable information on how to improve the selected articles
• Investigate the user interest in editing Wikipedia articles and how it changes over time

Two new features the data visualizations show:

• The benchmarks regarding the previous analysis
• The articles reviewed by the Wikipedia Community and by expert reviewers.

Results

Phase 0

Report of the initial status of the selected articles.

• 
  Network of incoming links
• 
  Network of outgoing links
• 
  Incoming and outgoing links
• 
  Pageviews
• 
  Features of the articles
• 
  Edits

Phase 1

Report of the provisional status of the selected articles considering the previous visual analysis. All datasets are open and available online.

• 
  Network of incoming links
• 
  Network of outgoing links
• 
  Incoming and outgoing links
• 
  Incoming and outgoing links (link from and to articles excluded)
• 
  Features of the articles
• 
  Features: comparison before-after of articles improved through the Wikipedia School Project
• 
  Features: comparison before-after of articles improved through assessment
• 
  Features: comparison before-after of articles improved by Wikipedia community
• 
  Features: comparison before-after of articles improved through edit-a-thon
• 
  Features: comparison before-after of articles improved by experts
• 
  Features: comparison before-after of articles improved through reassessment

Phase 2

Report of the final status of the selected articles considering the previous visual analysis. All datasets and protocols are open and available online.

• 
  Articles network in 2015
• 
  Articles network in 2017
• 
  Network of incoming links
• 
  Balance between incoming and outgoing links
• 
  Balance between incoming and outgoing links (except links to articles)
• 
  Pageviews
• 
  Comparison of articles network in August 2015 and August 2017

Findings

• The cluster of articles related to South Africa has been enhanced. That is due to the larger number of mutual links between articles belonging to South Africa topic. Some articles that links to the selected articles are the following:
  • Education in South Africa
  • Black Consciousness Movement
  • Scouts South Africa
  • Parliament of South Africa
  • New National Party (South Africa)
  • Cinema of South Africa

• Incoming links has slightly decreased. Outgoing links are increased. The two new articles created under the framework of WIkipedia Primary School project (Kaditshwene and Makhonjwa Mountains) have respectively 13 and 25 incoming links, 46 and 58 outgoing links.
• Significant reduction of issues related to articles under examination. Despite reviews, some issue on articles under examination still remain. Among them the following articles have issues:
  • San healing practices
  • Oliver Tambo
  • Youth culture
  • Wood
  • Gangster
  • Discrimination
  • Gender role
  • Health

• The number of features of the articles (within WIkipedia Primary School project we consider features the following: references, notes, images, see also) has grown. It emerges the trend to increase the features in the lacking articles and to filter out the relevant features in articles with a rich body. Articles with few features, among those under examination, are:
  • Children's Act, 2005
  • Day of Reconciliation
  • Heritage Day (South Africa)
  • Mapungubwe Museum
  • Water privatisation n South Africa

• Between 2014 and 2015 the total amount of editors has slightly decreased (from 845 to 773, -8.5%). The number of edits has slightly increased (from 1463 to 1481, +1,2%). It suggests a collective greater effort in contributing to the develop of Wikipedia articles. Among articles with none or very few edits in 2015 we found:
  • Children's Act, 2005
  • Frances Baard
  • Mapungubwe Museum
  • San healing practices

Documentation

• 
  Evaluation plan, September 2014.
• 
  Preliminary analysis, August 2015.
• 
  Intermediate analysis methodology, March 2016.
• 
  Intermediate analysis, March 2016
• 
  Final analysis, August 2017

Open datasets

All datasets are open and available online.

• Datasets

Related articles

• Research:Wikipedia Primary School SSAJRP programme

Credits

Giovanni Profeta is leading the evaluation of the project with a specific focus on information design.