Wikimedia Blog/Drafts/Article Feedback Update

This draft is for a Tech blog post that was published on Thu. Dec. 20 by 10:53 am PT, after review by Tilman. Here's the final post.

Article Feedback: New research and next steps

This feedback form engages readers to contribute to Wikipedia

How can we engage readers to contribute productively to Wikipedia?

Our recent work on Article Feedback v5 (AFT) provides new insights on that question. In this post, we'd like to share what we've learned by analyzing feedback and moderation activity -- as well as give a quick update on our next steps for this project, which is being developed by the Wikimedia Foundation's editor engagement team.

Article Feedback v5 aims to increase participation on Wikipedia (a strategic goal of the Wikimedia movement): this tool asks readers to suggest improvements to articles, then invites them to sign up and become editors. Another goal of this project is to help current editors improve article quality based on reader suggestions. (Learn more about Article Feedback.)

Last summer, we deployed this new tool on 10 percent of the English Wikipedia and we have been testing it extensively to evaluate its impact on readers and editors. Here's what we found so far.

Key findings

These highlights of our latest research on Article feedback are based on feedback and moderation data collected from September 7 to November 7, 2012 (more details can be found in these slides):

Many readers use this feature

Readers are already posting a lot of feedback with this tool -- an average of 4,100 posts per day from 2,800 daily unique readers, on just 10 percent of the English Wikipedia. At this rate, we project up to 900,000 feedback posts with comments per month once Article Feedback is deployed to 100 percent of English Wikipedia in 2013. About 98 percent of this feedback is from anonymous users who are not currently participating on Wikipedia. About 70 percent of the readers we surveyed liked this feedback tool after using it. Their responses suggest that it makes it easy for them to get involved and that they enjoy doing it.

This tool is converting readers into new editors

Article Feedback appears effective in getting new users to contribute to Wikipedia. For example, 2.7 percent of readers who post feedback go on to create a new account, after being invited to sign up. And 3 percent of these new users go on to edit articles within 24 hours from signing up. At this rate, we project several hundred thousand new registrations per year on the English Wikipedia -- resulting in many new contributors, which we hope can help reverse the current editor decline.

Useful feedback is buried under a lot of noise

In our first feedback evaluation study last spring, we asked 20 Wikipedia editors to blindly assess 900 random feedback posts for usefulness. About 40 percent of the feedback was found useful by at least two evaluators. This finding is consistent with this moderation data, which shows more negative than positive evaluations by community moderators. We also found that most of the moderation takes place on high traffic articles, which attract lower quality feedback. (To get a sense of what comments look like on a recently created article, check out the feedback page for the Sandy Hook Elementary School shooting article.)

Most feedback never gets moderated

Editor moderation activity is very low when compared to the volume of feedback posted every day. Less than 10% of all posts for a sample of the 100 most visited articles are moderated by registered editors within a month. In fact, less than 10% of feedback posted every single day receives any kind of moderation within a month, whether by readers or registered editors. In the current version of the tool, many editors can't easily find comments for articles they edit, an issue which could be addressed by making feedback more visible to editors on the article pages.

More results can be found in these slides, which summarize our analysis of feedback and moderation data. Live moderation data can also be viewed on this dashboard.

Next steps

Mockup of new moderation tools and filters under consideration.

Based on our these findings, we are now developing a final version of Article Feedback 5, which we plan to release in early 2013. Our goals for this release are to:

surface more good feedback
reduce the editor workload
improve software performance.

Here are some of the key features we are working on:

Better feedback filters

We are improving our filtering tools to automatically surface more good feedback and remove irrelevant comments. For example, the new feedback page will only feature comments that have been marked as helpful -- and feedback that has not yet been moderated will be listed in a separate filter. We are also creating more abuse filters to prevent users from posting inappropriate comments.

Simpler moderation tools

To reduce the editor workload, we are simplifying the feedback moderation tools. These new tools will enable moderators to quickly sort feedback into different groups, so that editors can focus on useful suggestions for improvements, without being distracted by comments that are not usable. We also plan to make useful feedback more visible to editors, through a special link on article pages.

Improved performance

We are refactoring our code to make this tool more scalable so it can support millions of comments with better database performance. This backend engineering work has taken longer than anticipated, in order to provide a solution that can be used by other projects.

Once these features have been developed, we plan to test them on 10 percent of the English Wikipedia, then release them to 100 percent in the first quarter of 2013. We expect more projects to deploy Article Feedback after this full release. For example, the German Wikipedia has already started a pilot to evaluate this tool on their site, and a similar pilot is under discussion on the French Wikipedia. For now, we invite you to try out Article Feedback for yourself on this sample article.

We would like to thank all the Wikipedians who have helped us design and develop Article Feedback this year. We look forward to deploying it widely early next year, to encourage more participation from readers. We hope this engagement tool can help sign up new contributors, to revert the editor decline and provide new ways for users to improve Wikipedia together.

Happy holidays!

Fabrice Florin, Product Manager
Dario Taraborelli, Senior Research Analyst
Oliver Keyes, Community Liaison
Wikimedia Foundation's Editor Engagement Team