Wikimedia Blog/Drafts/Comprehensive evaluation of Wiki contributors for recognition

This was a draft for a blog post that has since been published at https://blog.wikimedia.org/2014/04/29/comprehnsive-evaluation-of-wiki-contributors/

Title

Comprehensive evaluation of Wiki contributors for recognition

Body

The Telugu Wikipedia (Tewiki) community has recognized prolific contributors to Wikimedia projects with a special award called "Komarraju Lakshmana Rao Wikimedia Puraskaram"(KLRWP) on the tenth anniversary of Telugu Wikipedia on December 10th, 2013. The award is named after Komarraju Lakshmana Rao, the first modern encyclopedist in Telugu, perhaps the first among Indian languages (c.1912-13). It is a fitting tribute to honor Lakshmana Rao in the centennial anniversary of such a pioneering effort. Each award carried a citation and cash prize of 10,000 Indian Rupees (equaling about 166 USD as of Apr 2014). This blog post gives details about the process, its implementation and lessons learned from it.

Recognition on Wiki projects is usually done by awarding barnstars by peer contributors, when an editor crosses a significant edit count or makes substantial contribution to a project. Other approaches that have been tried included gift coupons for books, travel/accommodation and scholarships for events and cash prizes for best essays by students. There have been no monetary prizes for the people who have been contributing for several years. WMDE started Zedler-Preis für Freies Wissen (Zedler Prize for Free Knowledge) in 2012. The award categories include best article, best community project and best external project. The awards are in the form of scholarships for events or in cash for external entities.

The disadvantages that were observed with the traditional approaches were as followaed. The recognition process was dependent on a single person as opposed to a group. This led to an opinion that recognitions are dependent on number of edits, partiality, incomplete quantitative analysis of all factors. Person dependency also led to delay in recognizing the right candidates. Competitions only attracting new people who most often did not turn out to be active editors. Very few Wikimedians are getting recognized with scholarship for global Wikimania or national conferences. Competitions also have a tendency to make people support fellow Wikimedians less.

During the WikiConference India 2011, a simple process was pioneered by WMIN Cofounder and the then EC Member Arunram for a recognition called Noteworthy Wikimedian Recognition. This consisted of a simple proposal with few points about editors contributions, which was deliberated by a committee. The winners were recognized in the conference and an electronic barnstar was uploaded on their talk pages. We built upon this process by making the proposal more detailed and the evaluation more exhaustive.

A selection council of volunteer Wikipedians who had achieved some recognition in some form or another in the past were constituted, with the basic premise that they will not be eligible for the award. A template which allowed for a comprehensive view of various contributions like edits, images, policy development, projects, outreach, including contributions to sister language wiki projects etc was prepared and reviewed. The evaluation model was also developed with guidelines for minimum and maximum scores, with appropriate variation for specific Wikimedia project like Wiktionary and Wikisource. An example proposal was also provided using the contributions of Vyzasatya, a key and long time Telugu Wikipedian, who was on the team. Criticism from some members about the proposal and evaluation were addressed and the framework was improved where possible. Rules were developed for nominations, which could be done by the candidates or their peers. Endorsements of the candidature was also provided for in the nomination template.

Wide publicity was given in the village pumps, mailing lists and social media channels regarding the awards. For the benefit of wider Wikimedia community, we have provided relevant remarks on the nomination pages in English and allowed nominations in Telugu or English.

After a candidate was proposed by his/her peers, the nominee’s acceptance of the nomination was sought, as some of the wikimedians may not be interested in participating. We got 28 applications for the awards out of which 18 were complete, based on the acceptance criteria outlined in the initial announcement. Since many nominations did not provide enough detail or statistics, the selection council prepared a document with detailed statistics of contributions under each namespace along with relevant charts.

The selection council analyzed all proposals based on quantitative metrics like wiki statistics and qualitative assessment of each nominee's contribution and it's impact on Wiki 's growth and development. Factors considered in judging process include

Number of edits in article name space
Files contribution
Help documentation
Template, MediaWiki, Bot
WikiProjects, Support to Wiki Development
Support to the peer wikimedians, Wiki behavior
Online publicity
Physical publicity, participating in physical meetings
awareness & following the right Wiki process
Non Telugu Wiki projects contribution

The council held nine virtual meetings from the time of formation of the council until the finalization of the award. Various tools like Google docs, Wiki statistics (for gathering user’s edit counts, for graphics) , desktop sharing and Skype for conference calls were used.

As this is the debut year for the awards, the selection council decided to award 10 recipients, recognizing their contributions during the first decade of Telugu Wikipedia and its sister projects. In the order of final assessment, score, the awardees are Chaduvari, M Pradeep, Chava kiran, Veeven, Palagiri, Ravichandra, Ahmed Nisar, Veera.sj, JVRK PRASAD and Bhaskaranaidu. They were presented with cash prizes and certificates during the Telugu Wiki Ten Celebrationson on February 16th and 17th 2014. Pictures, citation remarks in Telugu and English [3 were prepared and shared widely.

‌Feedback was obtained from the participants and the community regarding the process. The process was appreciated for its comprehensiveness in evaluation of the contribution and transparency in its development and implementation. The selection council also became aware of the practices that aided or hindered the growth of Wikis. The former were highlighted in the analysis of each candidate's contributions. These included breadth and depth of contribution, knowledge of Wiki policies and courteousness in online discussion. The later included contributions without understanding the scope of Wiki project, rude behavior in online discussions, violation of Copyright and COI. These were consolidated as a group and shared with the community.

It was an arduous effort by the selection council members namely Ravi Vyzasathya, Arjuna Rao Chavala, A. Rajasekhar, T. Sujatha and Radhakrishna Arvapally. The council thanks Arjuna Rao Chavala, Selection council secretary for taking responsibility for this initiative. We also thank the Wikimedia Foundation for providing the necessary funds to support this award and Tewiki 10 Celebrations committee for their encouragement and support. We would like to thank all the people who participated in the nomination and proposal development process and also those who helped to improve the process by their feedback during its roll out.

Authors: Radhakrishna Arvapally, Arjuna Rao Chavala

Notes