Research:How does the community's reaction to newcomer work affect retention?

This question set is among the high level research priorities for the Wikimedia Summer of Research 2011. The full list is here, and this is categorized as "RQ1".

Community Reactions

Messaging

(RQ1.1) How does receiving no reaction from the community affect retention and future work? Preliminary work shows that, overall, receiving no feedback from the community during the newbie period does not necessarily detract new editors. However, there seems to have been some kind of historical shift around 2006. While in 2004-6 there was a negative correlation between length of ignored period and retention of new editors, after 2006, the trend was reversed, and editors who were ignored longer tended to be retained longer than those who received immediate attention from the community.

Collaboration

(RQ1.2) What proportion of newbie edits are significantly refactored by other editors (rather than being reverted)?

(RQ1.3) How does a refactoring edit (that modifies, but does not revert the newbie’s contribution) affect retention and future work?

(RQ1.4) How do other editor’s contributions to a newbie’s created article affect retention? examines newbie reactions to other people editing the page they created. Newbies have a negative reaction to any interaction with their article by other editors. Whether they are adding or removing content, the effect is the same--newbies will do less work. Surprisingly, adding content to a newbie’s article is more demotivating than removing content. They didn’t look specifically at page deletions.

Reverting

(RQ1.5) How significant is the effect of reverts toward retention and future work? suggests that being reverted strongly effects survival (continuing to edit for at least the next month) of ~-15% per revert. Of the editors who continue to edit, their quantity of work will also increase by .23 standard deviations (quite a lot), so it appears that editors improve the average quality of their work after being reverted.

(RQ1.6) Have reverts towards good faith editors increased as a percentage? Yes . Only looked at reverts to new editors in general (good faith is not checked).

(RQ1.7) Do users notice that they are being reverted/warned? shows that editors react differently depending on who reverted them. Editors reverted by anons are less strongly affected than editors reverted by other named editors.

(RQ1.8) Is a newbie particularly effected by having their first revision reverted? Suggestion: Of the editors whose first edits were reverted, yet they stayed, what is their subsequent editing behavior in the next 30 days, compared to editors that did not get reverted? Plot the number of first edits that are reverted vs. not reverted from anon and registered; plot one-month retention after.

(RQ1.9) Are newbies getting reverted more as the core articles are being completed? Yes . Since length of article is highly correlated with quality/completeness, we can use length as a proxy. The longer an article that a newbie edits, the more likely they’ll be reverted. This is concerning because the articles that newbies edit have been growing in length on average.

(RQ1.10) How do newbies react to policy enforcement (eg. invoking WP:XXX) and who is citing policy to newbies? suggests that new users cite policy because it strengthens their feeling of belonging to the community. However policy citations by these newcomers are likely to be more aggressive towards new editors (and have a stronger impact on retention) than those made by veterans.

Page deletion

(RQ1.11) What is the impact of an article being nominated for deletion, vs. being actually deleted, on an editor's retention?

(RQ1.12) Have deletion nominations towards newly created articles increased as a percentage? Yes . Less notable articles are being created and more of them are being deleted. There was a dip in 2006 when registering an account was required before creating a page, but the system has recovered and > 40% of articles that were created in 2009 have been deleted. Additionally, while both the number of deletions and the number of deletion notifications to new users has increased, the number of new editors participating in deletion discussion has been steadily dropping over the years. (Results here).

(RQ1.13) What percentage of these deletions occur for acceptable articles, ones that don't meet the deletion criteria; Or for articles that would probably merit deletion via AFD but are incorrectly deleted via speedy deletion? shows that there are many deletions that probably shouldn’t happen and there are admins who can be picked out for having a strong bias. Biased AFD closing editors will account for a striking ~35% error rate. Deletion discussions with more editors involved make better decisions. Canvassing for input from results in a worse decision on average.

(RQ1.14) How has the distribution of deletion activities changed over time?

(RQ1.15) Has there been more or less activity in specific deletion areas such as speedy (CSD), Articles for Deletion (AFD), proposed deletions (PROD) or Miscellany for Deletion (MFD)?

(RQ1.16) Who are the most active new page patrollers and how do they work? Preliminary results here.

Introductions to Wikipedia

(RQ1.17) How are new users welcomed to the project? Does the tone, links, and style of welcome messages affect further participation?

Clustering

(RQ1.18) Does editing popular articles with high levels of activity from experienced editors have a negative impact on the likelihood that a new editor is retained?