Grants:IEG/Revision scoring as a service/Timeline

This project is funded by an Individual Engagement Grant

This Individual Engagement Grant is renewed

Timeline for Revision scoring as a service

Timeline	Date
Project kickoff	8 December 2014
Working trainable classifier. Project init for hand-coder. Language descriptions for en, pt, tr & az. High accuracy classification of reverts. MVP API on Wikimedia Labs.	6 March 2015
Working hand-coder & classifiers trained on "damage" and "good-faith". Volunteer hand-coders for en, pt, tr & az. API performance is fast enough to classify edits in real time.	1 June 2015

Monthly updates

Please prepare a brief project update each month, in a format of your choice, to share progress and learnings with the community along the way. Submit the link below as you complete each update.

December

Hey folks. This updated is going to be brief. We have weekly updates posted on our project talk page: Research talk:Revision scoring as a service. Look there for more detail.

Last month, we worked with focus on building up a scorer management system. Everyone on the project team is up to speed and has at least had a hand in merging a pull request. See https://github.com/halfak/Revision-Scoring for our main repo. Right now, we have one powerful classifier working (.85 AUC for predicting reverts in Enwiki).

We've also started work on a revision hand-coding tool. See https://github.com/he7d3r/mw-gadget-QualityCoding for progress there.

Language features are complete for Enwiki and Ptwiki, but we're still working on getting the badwords list fleshed out for Trwiki.

We've also started work on the system that will serve scores -- which we have decided to refer to as ORES -- The Objective Revision Evaluation Service. See https://github.com/halfak/Objective-Revision-Evaluation-Service.

January

Brief updates again. See our project talk page for weekly updates.

In January, we did a lot of work to make the revscoring library easier to use -- including registering it in the python package index, so you can install it with "pip install revscoring". During this time, we substantially improved out ability to make predictions and fleshed out several ipython notebooks demonstrating the fitness of revert classifiers. We presented on the this revscoring library at the January Metrics Meeting (video, slides).

We also kicked off the ORES project for serving classifiers via an API. By the end of the month, we had several different classification strategies tested and several convenience scripts generated in order to gather feature sets and train/test classifiers.

We also spent a decent amount of effort extending upstream projects. For example, we've pushed a large set of improvements to mediawiki-utilities and had pull requests merged to python's natural language toolkit.

Finally, we extended our set of mockups of the Revision Coding gadget system to include a new progress tracking format and revcoder homepage.

February

This month, we successfully deployed a ORES revision scoring service prototype and it is *fast*. The prototype is open to external testing. See http://ores.wmflabs.org/scores/ptwiki?models=reverted&revids=4567890 and http://ores.wmflabs.org/scores/enwiki?models=reverted&revids=4567890 for sample queries for enwiki and ptwiki revert scores.

In order to achieve this milestone, we performed substantial work to improve the performance of the revscoring library. We also standardized naming and released two major versions of the revscoring software. We developed a generalized web service for hosting scoring models and deployed it to a server that we configured in WMF labs.

During this time, we continued our outreach work with a report in the English Wikipedia signpost (see en:Wikipedia:Wikipedia_Signpost/2015-02-18/Special_report) that was translated to Portuguese Village pump pt:Wikipédia:Café dos programadores#Serviço de pontuação de edições and Turkish Wikipedia tr:Vikipedi:Köy_çeşmesi_(ilginize)#Değişiklik Değerlendirme Projesi.

For further updates, please see Research talk:Revision scoring as a service.