Research:Newsletter/2014/July

Vol: 4 • Issue: 7 • July 2014 [contribute] [archives]

Shifting values in the paid content debate; cross-language vandalism detection; translations from 53 Wiktionaries

With contributions by: Piotr Konieczny, Maximilian Klein, Heather Ford, and Han-Teng Liao

Understanding shifting values underlying the paid content debate on the English Wikipedia

See related Signpost content: "Extensive network of clandestine paid advocacy exposed", "With paid advocacy in its sights, the Wikimedia Foundation amends their terms of use"

Reviewed by Heather Ford

Kim Osman has performed a fascinating study on the three 2013 failed proposals to ban paid advocacy editing in the English language Wikipedia. Using a Constructivist Grounded Theory approach, Osman analyzed 573 posts from the three main votes on paid editing conducted in the community in November, 2013. She found that editors who opposed the ban felt that existing policies of neutrality and notability in WP already covered issues raised by paid advocacy editing, and that a fair and accurate encyclopedia article could be achieved by addressing the quality of the edits, not the people contributing the content. She also found that a significant challenge to any future policy is that the community 'is still not clear about what constitutes paid editing'.

Osman uses these results to argue that there has been a transition in the values of the English language Wikipedia editorial community from seeing commercial involvement as direct opposition to Wikipedia's core values (something repeated at the institutional level by the Wikimedia Foundation and Jimmy Wales who see a bright line between paid and unpaid editing) to an acceptance of paid professions and a resignation to their presence.

Osman argues that the romantic view of Wikipedia as a system somehow apart from the commercial market that characterized earlier depictions (such as those by Yochai Benkler) has been diluted in recent years and that sustainability in the current environment is linked to a platform's ability to integrate content across multiple places and spaces on the web. Osman also argues that these shifts reflect wider changes in assumptions about commerciality in digital media and that the boundaries between commercial and non-profit in the context of peer production are sometimes fuzzy, overlapping and not clearly defined.

Osman's close analysis of 573 posts is a valuable contribution to the ongoing policy debate about the role of paid editing in Wikipedia and will hopefully be used to inform future debates.

"Pivot-based multilingual dictionary building using Wiktionary"

Reviewed by Maximilian Klein (talk)

Straight edges represent translation pairs extracted directly from the Wiktionaries. The pair guild–breaslawas found via triangulating.

To build multilingual dictionaries to and from every language is combinatorially a lot of work. If one uses triangulation–if A means B, and B means C, then A means C (see figure)–then a lot of the work can be done by machine. A large closed-source effort did this in 2009, but a new paper by Ács defends "while our methods are inferior in data size, the dictionaries are available on our website". Their approach used the translation tables from 53 Wiktionaries, to make 19 million inferred translations more than the 4 million already occurring in Wiktionary. The researchers steered clear of several classical problems like polysemy, one word having multiple meanings, by using a machine learning classifier. The features used in the classifier were based on the graph-theoretic attributes of each possible word pair. For instance, if two or more languages can be an intermediate "pivot" language for translation, that turned out to be a good indicator of a valid match. In order to test the precision of these translations, manual spot checking was done and found a precision of 47.9% for newly found word-pairs versus 88.4% for random translations coming out of Wiktionary. As for recall, which tested the coverage of a collection of 3,500 common words, 83.7% of words were accounted for by automatic triangulation in the top 40 languages. That means that right now if we were to try and make a 40-language pocket phrasebook to travel around most of the world just using Wiktionary, about 85% of the time there would be a translation, and it would be between 50-85% correct.

This performance would likely need to increase before any results could be operationalized and contributed back into Wiktionary. However, given the fact that the code used to parse and compare 43 different Wiktionaries was also released on GitHub, that goal is a possibility. It's yet another testament to the open ecosystem to see a Wikimedia project along with Open Researcher efforts make a resource to rival a closed standard. While Ács' research isn't the holy grail of translation between arbitrary languages, it cleverly mixes established theory and open data, and then contributes it back to the community.

"Cross Language Learning from Bots and Users to detect Vandalism on Wikipedia"

Reviewed by Han-Teng Liao (talk)

A new study by Tran and Christen is the latest example of academic research on vandalism detection which has been developed over the years in the context of the PAN workshop, where researchers develop both corpus data and tools to uncover plagiarism, authorship, and the misuse of social media/software. This work should be of interests to both researchers and Wikipedians because of (a) the need to detect vandalism and (b) the interesting question whether such vandalism-fighting data and tools are transferable or portable from one language version to another. Both the vandalism-fighting corpus and tools have both practical and theoretical implications for understanding the cross-lingual transfer in knowledge and bots.

In 2010 and 2011, Wikipedia vandalism detection competitions were included by the PAN as workshops. It started with Martin Potthast's work on building the free-of-charge PAN Wikipedia vandalism corpus, PAN-WVC-10 for research, which compiled 32452 edits based on 28468 Wikipedia articles, among which 2391 vandalism instances were identified by human coders recruited from Amazon's Mechanical Turk. In 2011, a larger crowdsourced corpus of 30,000+ Wikipedia edits is released in three languages: English, German, and Spanish, with 65 features to capture vandalism.

Based on even larger datasets of over 500 million revisions across five languages (en:English, de:German, es:Spanish, fr:French, and ru:Russian), Tran & Christen's latest work adds to the efforts by applying several supervised machine learning algorithms from the Scikit-learn toolkit, including Decision Tree (DT), Random Forest (RF), Gradient Tree Boosting (GTB), Stochastic Gradient Descent (SGD) and Nearest Neighbour (NN).

What Tran & Christen confirm from their findings is that "distinguishing the vandalism identified by bots and users show statistically significant differences in recognizing vandalism identified by users across languages, but there are no differences in recognizing the vandalism identified by bots" (p.13) This demonstrates human beings can recognize a much wider spectrum of vandalism than bots, but still bots are shown to be trainable to be more sophisticated to capture more and more nonobvious cases of vandalism.

Tran & Christen try to further make the case for the benefits of cross language learning of vandalism. They argue that the detection models are generalizable, based on the positive results of transferring the machine-learned capacity from English to other smaller Wikipedia languages. While they are optimistic, they acknowledge such generalization has at best been proven among some of the languages they studied (these languages are all Roman-alphabet-based languages except for Russian), and the poor performance of the Russian language model. Thus, Tran & Christen rightly point out the need for research on non-English and especially non-European language versions. They also recognize that many word based features are no longer useful for some languages such as Mandarin Chinese, because of tokenization and other language-specific issues.

Tran & Christen call for next research projects to include languages such as Arabic and Mandarin Chinese to complete the United Nations working set of languages. It will be interesting to see how such research projects can be executed and how the greater Wikipedia research and editor community can help and/or use such research efforts.

Readers' interests differ from editors' preferences

Reviewed by Piotrus.

A conference paper titled "Reader Preferences and Behavior on Wikipedia" deals with the under-studied population of Wikipedia readers. The paper provides a useful literature review on the few studies about reading preference of that group. The researchers used publicly available page view data, and more interestingly, were able to obtain browsing data (such as time spend by a reader on a given page). Since such data is unfortunately not collected by Wikipedia, the researchers obtained this data through volunteers using a Yahoo! toolbar. The authors used Wikipedia:Assessment classes to gauge article's quality.

The paper offers valuable findings, including important insights to the Wikipedia community, namely that "the most read articles do not necessarily correspond to those frequently edited, suggesting some degree of non-alignment between user reading preferences and author editing preference". This is not a finding that should come as much surprise, considering for example the high percentage of quality military history articles produced by the WikiProject Military History, one of the most active if not the most active wikiproject in existence - and of how little importance this topic is to the general population. Statistics on topics popularity and quality of corresponding articles can be seen in Table 1, page 3 of the article. Figure 1 on page 4 is also of interest, presenting a matrix of articles grouped by popularity and length. For example, the authors identify the area of "technology" as the 4th most popular, but the quality of its articles lags behind many other fields, placing it around the 9th place. It would be a worthwhile exercise for the Wikipedia community to identify popular articles that are in need of more attention (through revitalizing tools like Wikipedia:Popular pages, perhaps using code that makes WikiProject popular pages listing work?) and direct more attention towards what our readers want to read about (rather than what we want to write about). Finally, the authors also identify different reading patterns, and suggest how those can be used to analyze article's popularity in more detail.

Overall, this article seems like a very valuable piece of research for the Wikipedia community and the WMF, and it underscores why we should reconsider collecting more data on our readers' behavior. In order to serve our readers as best as we can, more information on their browsing habits on Wikipedia could help to produce more valuable research like this project.

Wikipedia from the perspective of PR and marketing

Reviewed by Piotrus.

An article in "Business Horizons", written in a very friendly prose (not a common finding among academic works), looks at Wikipedia (as well as some other forms of collaborative, Web 2.0 media) from the business perspective of a public relations/marketing studies. Of particular interest to the Wikipedia community is the authors goal of presenting "the three bases of getting your entry into Wikipedia, as well as a set of guidelines that help manage the potential Wikipedia crisis that might happen one day." The authors correctly recognize that Wikipedia has policies that must be adhered to by any contributors, though a weakness of the paper is that while it discusses Wikipedia concepts such as neutrality, notability, verifiability, and conflict of interest, it does not link to them. The paper provides a set of practical advice on how to get one's business entry on Wikipedia, or how to improve it. While the paper does not suggest anything outright unethical, it is frank to the point of raising some eyebrows. While nobody can disagree with advice such as "as a rule of thumb, try to remain as objective and neutral as possible" and "when in doubt, check with others on the talk page to determine whether proposed changes are appropriate", given the lack of consensus among Wikipedia's community on how to deal with for-profit and PR editors, other advice such as "maximize mentions in other Wikipedia entries" (i.e. gaming WP:RED), "be associated with serious contributors...leverage the reputation of an employee who is already a highly active contributor... [befriend Wikipedians in real life]", "When correcting negative information is not possible, try counterbalancing it by adding more positive elements about your firm, as long as the facts are interesting and verifiable", "...you might edit the negative section by replacing numerals (99) with words (ninety-nine), since this is also less likely to be read. Add pictures to draw focus away from the negative content" might be seen as more controversial, falling into the gaming the system gray area. The "Third, get help from friends and family" section in particular seems to fall foul of meatpuppetry.

In the end, this is an article worth reading in detail by all interested in the PR/COI topics, though for better or worse, the fact that it is closed access will likely reduce its impact significantly. On an ending note, one of the two article's co-authors has a page on Wikipedia at Andreas Kaplan, which was restored by a newbie editor in 2012, two years after it's deletion, has been maintained by throw-away SPAs, and this reviewer cannot help but notice that it still seems to fail Wikipedia:Notability (academics)...

"No praise without effort: experimental evidence on how rewards affect Wikipedia's contributor community"

Reviewed by Piotrus.

In 2012, the authors of this paper have given out over a hundred barnstars to the top 1% most active Wikipedians, and concluded that such awards improve editors productivity. This time they repeated this experiment while broadening their sample size to the top 10% most active editors. After excluding administrators and recently inactive editors, they handed out 300 barnstars "with a generic positive text that expressed community appreciation for their contributions", divided between the 91st–95th, 96th–99th, and 100th percentiles of the most active editors (this corresponds to an average of 282, 62 and 22 edits per month) and then tracked the activity of those editors, as well as of the corresponding control sample which did not receive any award. The experiment was designed to test the hypothesis that less active contributors will be responsive to rewards, similar to the most highly-active contributors from the prior research.

The authors found, however, that rewarding less productive editors did not stimulate higher subsequent productivity. They note that while the top 1% group responded to an award with an increase in productivity (measured at a rather high 60% increase), less productive subjects did not change their behavior significantly. The researchers also noted that while some of the top 1% editors received an additional award from other Wikipedians, not a single subject from the less active group was a recipient of another award.

The researchers conclude that "this supports the notion that peer production’s incentive structure is broadly meritocratic; we did not observe contributors receiving praise or recognition without having first demonstrated significant and substantial effort." While this will come as little surprise to the Wikipedia community, their other observation - that outside the top 1% of editors, awards such as barnstars have little meaningful impact - is more interesting.

Further, the authors found that while rewarding the most active editors tends to increase their retention ratio, it may counter-intuitively decrease the retention ratio of the less active editors. The authors propose the following explanation: "Premature recognition of their work may convey a different meaning to these contributors; instead of signaling recognition and status in the eyes of the community, these individuals may perceive being rewarded as a signal that their contributions are sufficient, for the time being, or come to expect being rewarded for their contributions." They suggest that this could be better understood through future research. For the community in general, it raises an interesting question: how should we recognize less active editors, to make sure that thanking them will not be taken as "you did enough, now you can leave"?

Briefly

Wikipedia assignments improve students' research skills

It is refreshing to see a continuing and growing stream of academic works endorsing various aspects of teaching with Wikipedia paradigm. A study of eleven students "enrolled in a semester-long academic literacy course in a preparatory program for study at an Australian university... showed an educationally statistical improvement in the students’ research skills, while qualitative comments revealed that despite some technical difficulties in using the Wikipedia site, many students valued the opportunity to write for a ‘real’ audience and not just for a lecturer."

A split in the growing field of Chinese-language Wikipedia research

A blog post by Han-Teng Liao (廖漢騰) presents an interesting exploratory overview of a Chinese language research on Wikipedia. The findings suggest that Chinese-language scholars and academic publication outlets are increasingly doing research in the field of Wikipedia studies; however there's "a divide between mainland Chinese academic sources/search results on one hand, and Hong Kong/Taiwanese ones on the other." The reason for this seems to be primarily technical, as scholars from different regions seem to publish in different outlets, which in turn are not indexed in the academic search engines preferred by those from other region.

Other recent publications

A list of other recent publications that could not be covered in time for this issue – contributions are always welcome for reviewing or summarizing newly published research.

• "Uneven Openness: Barriers to MENA [Middle East/North Africa] Representation on Wikipedia" (blog post)
• " Detecting epidemics using Wikipedia article views: A demonstration of feasibility with language as location proxy"
• "The Reasons of People Continue Editing Wikipedia Content - Task Value Confirmation Perspective"
• "Circling the Infinite Loop, One Edit at a Time: Seriality in Wikipedia and the Encyclopedic Urge"
• "Identifying Duplicate and Contradictory Information in Wikipedia"
• "The impact of elite vs. non-elite contributor groups in online social production communities: The case of Wikipedia"
• "What do we Think an Encyclopaedia is?" From the abstract: "Based on survey and interview research carried out with publishers, librarians and higher education students, [this article] demonstrates that certain physical features and qualities are associated with the encyclopaedia and continue to be valued by them. Having identified these qualities, the article then explores whether they apply to three incidences of electronic encyclopaedias, Britannica Online, The Stanford Encyclopedia of Philosophy and Wikipedia."
• "Crowdsourcing Knowledge Interdiscursive Flows from Wikipedia into Scholarly Research". From the abstract: "using a dataset collected from the Scopus research database, which is processed with a combination of bibliometric techniques and qualitative analysis [this article finds] that there has been a significant increase in the use of Wikipedia as a reference within all areas of science and scholarship. Wikipedia is used to a larger extent within areas like Computer Science, Mathematics, Social Sciences and Arts and Humanities, than in Natural Sciences, Medicine and Psychology."
• "How Readers Shape the Content of an Encyclopedia: A Case Study Comparing the German Meyers Konversationslexikon (1885-1890) with Wikipedia (2002-2013)"

References

Supplementary references and notes:

Wikimedia Research Newsletter
Vol: 4 • Issue: 7 • July 2014
About • Subscribe: Email • [archives] • [Signpost edition] • [contribute] • [research index]