Research:Impact of The Wikipedia Adventure on new editor retention

Created

18:23, 17 April 2017 (UTC)

Contact

Sneha Narayan

Northwestern University

Collaborators

Jacob Orlowitz

Wikimedia Foundation

Jonathan T. Morgan

Wikimedia Foundation

Benjamin Mako Hill

University of Washington

Aaron Shaw

Northwestern University

Duration:  2014-November – 2016-October

---

Research:Projects

This page documents a completed research project.

This project presents an analysis of the impact of The Wikipedia Adventure (TWA), a gamified Wikipedia tutorial, on new editor retention. It combines survey results from the original TWA pilot study with a extended analysis of the log data collected for that study.

Methods

We evaluate TWA in two ways. We perform a survey that assesses how new users perceive the system’s design and tone. We then conduct a randomized controlled field experiment in which we invite new Wikipedia editors to use the system and measure its effect on their subsequent contributions using multiple parametric and nonparametric techniques. One of these techniques is two-stage least squares regression which we use to estimate the effect of playing TWA (conditional on having been invited to do so) on the number of future contributions.

For the field experiment, we present three hypotheses. Compared with the control condition (did not receive an invite to TWA), new editors who are invited to play TWA and who do so will make:

1. an increased number of subsequent contributions overall
2. an increased number of subsequent contributions to talk pages
3. subsequent contributions of greater average quality

Results

The findings from this study were published in the proceedings of the 2016 ACM conference on Computer Supported Cooperative Work. A pre-print of the paper is available.

Survey results suggest that respondents found the gamified aspects of the tutorial rewarding and engaging and agreed that a tutorial that provides a broad overview of editing should be shared with new editors on Wikipedia.

However, results from the field experiment indicate that playing TWA did not have a long-term, positive impact on new editor retention. While the treatment group (those who were invited + played TWA) contained a handful of users who showed extremely high outcomes in terms of subsequent total edits and talkpage edits compared to the control group, there was no significant difference between the two groups for subsequent talkpage edits or edit quality.

The results of the comparison in total edits between the treatment and control groups indicates that editors who played TWA made significantly fewer subsequent edits than editors who were not sent an invite and did not play the game.

While to study design does not provide a definitive answer to the question of why TWA was ineffective, we discuss a range of possible factors, including the specific design of TWA, the limits of gamification in the context of self-directed learning endeavors, cultural mismatches between TWA and other aspects of the new editor experience on Wikipedia, and the limits of institutional-style socialization within voluntary peer production settings. These discussion points, as well as the study's limitations, are laid out in greater detail within the research manuscript.