Research talk:The Rise and Decline

http://www-users.cs.umn.edu/~halfak/summaries/The%20Rise%20and%20Decline/

Signpost summary

I thought I'd take a stab at further condensing this already condensed write-up so it can fit into a one-page Signpost summary. Below is my attempt to sum up the whole thing in a bulleted (highly glib :D) outline. I took some liberties on the recommendations; feel free to edit.

Rise and Decline explained

• Decline: since 2007, there's been a slow and steady decline in new users entering the project. But why?
• Maybe it's because today's newbies suck. We explored this option through blind handcoding of thousands of newbie contributions from 2002 to the present.
• Result: turns out, not so much. Newbies are pretty much the same quality as they've always been. So why aren't they sticking around?

1. Bots and semi-automated editing tools: they're impersonal and make experienced users less likely to stop and talk to or help out the newbie they reverted. The newbie isn't mentored, doesn't learn from his/her mistakes, and either gets blocked or leaves.
2. Policy: it's become more calcified and difficult to change, and experienced users control its interpretation and development.

• Recommendations: 1) Make tools that optimize for good-faith newbies, not blatant vandals, and more good-faith newbies will be retained. 2) Make it easier for new users to participate in policy discussions and contribute to the social norms of the community. Restore the spirit of inventiveness, experimentation, and creativity that was present in the early days of Wikipedia, and tone down the bureaucracy.

What do you think? I think it would be good to keep this short, sweet, and to the point, and link to the Meta page and full paper for those who want to read further. Maximizes your chances of getting it to stick in people's eyeballs and brains :) Maryana (WMF) (talk) 20:34, 6 September 2012 (UTC)

Bureaucracy is getting better

The paper does contain good news though:

---

To explore Hypothesis: Norm formalization & calcification, we first looked for changes in the rate of new policy creation following the introduction of a structured proposal process in 2005. Figure 8 shows that growth of policies and guidelines began to slow in 2006, just as Forte (2009) reports. The results from our analysis of new policy/guideline proposals show that the number of new policy proposals accepted via this process peaked in 2005 at 27 out of 217 (12% acceptance). 2006 saw an even higher number of proposed policies, but lower acceptance with 24 out of 348 proposals accepted (7% acceptance). From 2007 forward, the rate at which policies are proposed decreases monotonically down to a mere 16 in 2011 while the acceptance rate stays steady at about 7.5%.

---

In other words, it would seem that en.wiki, contrary to popular belief, has developed a good immune system against bureaucracy norms expansion.

The paper is actually of little use in this part IMHO, because:

1. we already know [from previous research] that users who joined in 2005/2006 are still disproportionately active in most community processes like deletion discussions and so on,
2. everybody knows that to influence how the wiki is run it's more effective to change a single word in an important policy than to establish ten new policies.

As for (1), I doubt the Wikipedia thought police is keeping newcomers out of discussions, but one can make them look so hard that newbies won't participate. However, it.wiki recently switched from the established vote-system for deletion to a discussion system as en.wiki's, and a year of data for the "new" system seems to prove that it increased the words spent and drove away old/unexperienced editors (with 3+ years or 51-5000 edits), while newcomers resisted, presumably to defend their own articles. https://toolserver.org/~mauro742/liste/pdc_stats.csv <https://it.wikipedia.org/wiki/Wikipedia:Elenchi_generati_offline/Richieste/Archivio/2011#Lavoro_per_le_PdC>

Nemo 4 January 2013

Causality

Readers of this work tend to vastly oversimplify its own conclusions, which were: «rejection of contributions, especially for desirable newcomers, has substantially affected the decline». In particular, some tend to claim this work proved that unjustified rejection of valid contributions caused the decrease in editor retention. But the paper correctly doesn't state this, though it tries hard to prove it.

No attempt is in fact made to identify/claim a causality, beyond the mere coincidence/correlation: «while rejection is a strong negative predictor for survival, there are other independent effects over time that are reducing the rate of survival of newcomers». So it's possible that the good faith "non-golden" newcomers would have been lost nevertheless; or that they were brought away by some external factor; or that they were lost due some internal factor, coinciding with those examined but different (e.g. not reversion by a tool, but the way such reversion is presented); or even opposite from those examined (e.g., because the rules were relaxed and rejection of non-golden contributions decreased, editors learnt less and didn't manage to become productive).

1. The chosen features are not orthogonal enough to a number of other factors which are seemingly excluded from possible causes. For instance: "year" of arrival correlates to events in that year (like the infamous arise of Facebook; or internet penetration); reversion by a tool correlates with extreme cluelessness (if even a bot detects your edit as unproductive, the edit was probably very silly).
2. The logistic regression coefficients found are not quite conclusive. Only one of them varies of more than 2 standard deviations between the two classes (all vs. golden + good faith), and it's "deleted" (from -1.45 to -0.8). Not a very scientific consideration, perhaps, but worth being careful IMHO.
3. Relatedly, why are blocks not one of the features?

There are, however, some unacknowledged weak spots in the work, IMHO.

1. Deleted edits are included, but normal users have no access to deleted contributions. It's not explained how access to those was achieved, nor how selection was performed, nor what criteria were used to classify deleted contributions as desirable.
2. Figure 2 shows a substantial decrease in "golden" newcomers (i.e. those who succeed in respecting content guidelines), from 60+ % to less than 40 %, but this is given less importance than figure 3 which shows a smaller increase in "rejection rate", from ~10 to ~20 %. With the same data, one could have written «English Wikipedia proved to be very welcoming and understanding to newcomers: "clueful" newcomers frequency decreased of some 30 percentage points, but only 10 percentage points more newcomers saw their contribution outright rejected».
3. Figure 3 mixes "good faith" and "golden" editors and is therefore unable to tell us what productive editors we may have lost. There are good faith editors who are nevertheless unable to contribute productively to a project: if such a group increases in size, the rejection across the board will look higher even if it's, in fact, proportionally stable for each class. In other words, figure 3 is comparing apple with oranges.

A number of lines of research, if explored, could potentially bring counterexamples, disproving the claim. For instance:

• finding one single language edition of Wikipedia where there was no increased "rejection of contributions, especially for desirable newcomers", but there was a decrease in retention anyway;
• different classification/selection of deleted edits/pages;
• finding a way to identify "productive" newcomers, i.e. not merely good faith but also able to later improve and respect content guidelines;
• including the contributions blocked by AbuseFilter or other means which stop obviously bad edits before they reach the revision table (March 2009 corresponds to an increase in figure 3).

Well, I think this is enough for now. :) --Nemo 13:04, 12 December 2014 (UTC)

Raw data?

Is rawer data for Research:The Rise and Decline#The decline in good-faith newcomers available anywhere, EpochFail, in a table on Commons or Wikidata or something? Just the absolute numbers in each category or some such? I'd like to learn to use some of these data tools, and this seems like a really good dataset to work on. Thank you very much for making it. HLHJ (talk) 05:28, 21 November 2018 (UTC)

Even better, Maximilianklein has labeled a more recent labeled dataset. If you're looking for historical data, it'll take me some time to dig it up. But if recent data is more relevant (which I'm guessing it is) then, I think Max can help. --EpochFail (talk) 15:44, 21 November 2018 (UTC)

Thanks @EpochFail:. Hi @HLHJ:, yes I've been labelling a new dataset. A live way to get the lastest version is detailed in this ipython notebook on github, its a tad rough, but if you can get to df = pd.DataFrame.from_dict(completed_row_oriented), then you'll have it. If you need any help with the code, or would like to contribute to the dataset, or want to talk about research, then contact me (either on-wiki, or you can email me by going to my talk page and use the "email user" button in the sidebar). Maximilianklein (talk) 16:57, 21 November 2018 (UTC)

Thank you, EpochFail, Maximilianklein. I was interested in the 2007-ish transition, specifically, as I've been getting into discussions about it on the Wishlist (and even reusing the graphs from this page, as for instance here). It's an important dataset for editor retention, so I expect this relevance will not be a one-off. HLHJ (talk) 04:24, 22 November 2018 (UTC)

Ah well @HLHJ:, if you are interested in the actual 2007 study and data then I'm afraid my usefulness is limited. I could answer current questions about the state of newcomers etc. if that ever happened to be useful to you. Maximilianklein (talk) 23:53, 22 November 2018 (UTC)

Thank you, Maximilianklein, I really appreciate that offer and may well take you up on it. 01:09, 23 November 2018 (UTC)