Research:Productive new editor

Standard metric

Productive new editor

Specification

A ${\text{productive new editor}}(n,t)$ is a new editor who completes at least $n$ productive edit(s) within $t$ time since registration ($T$).

WMF Standard

• $n$ = 1 productive edit
• $t$ = 1 day

Measures

Editor productivity

Aliases

Productive newcomer

Related metrics

Newly registered user • New editor • Productive edit

Repository

Demo: https://github.com/halfak/Productive-new-editor-demonstration

Status

completed

Productive new editor is a standardized user class used to measure the number of first-time editors in a wiki project over time who make productive contributions. It's used as a proxy for editor productivity, and to a lesser extent, editor activation. A "productive new editor" is a new editor who saves revisions to content namespace pages that are not reverted.

Discussion

Excluding edits to deleted content

Spammers and other non-productive new editors tend to create articles that are non-productive and those articles tend to be deleted rather than the edits to the articles being reverted (and therefore excluding them from the productive edit criteria). Edits to articles that are deleted by the end of a new editor's first week since registration are not included in counts of productive edits.

The n productive edits threshold

Like choosing an $n$ for any metric based on counts (e.g. new editor and active editor), choosing a threshold is somewhat arbitrary. Choosing a higher threshold will result in a smaller proportion of newly registered users being considered productive.

The t time cutoff

There are a few ways that the timespan for identifying productive edits can be drawn. The two most common ways are based on time bounds and events. A time-bounded approach is based on the use of some $t$ cutoff to limit observations to a certain amount of time after a user registered their account. An event-based approach will use some event as the starting point to count user contributions. Another candidate time-span includes edits that a newcomer performed in their first edit session. Since productive new editor qualifies the activity of a new editor we set $t=1{\text{ day}}$, which effectively makes the class of productive new editors a proper subset of new editors. We analyze the effect of choosing a different value for $t$ below.

Time to revert cutoff

Because a revert can theoretically occur years after the original edit, r en:Censoring_(statistics) everts are only counted if they occurred within 48 hours of the original edit. For more details, see Research:Revert#Time to revert cutoff.

Limitations

• This metric represents productivity as a binary attribute of a user, it does not measure how productive a new editor is. New editors who make many productive edits and contribute substantial amounts of content will look identical (under this metric) to new editors who fix a few typos.
• The most clever vandalism/vandals may go unnoticed for more than 48 hours.

Lag time

Generation of this metric will need to be delayed by $t+2{\text{ days}}$ after users' registration dates in order to allow $t{\text{ days}}$ days for newly registered users to make edits and an additional $2{\text{ days}}$ for other editors to have a chance to revert them. In the case of the WMF Standard parameterization, this works out to $1+2=3{\text{ days}}$.

Analysis

Besides the variables describing a productive edit, there are two variables used in this metric:

• The value of $n$
• The value of $t$

Given that the raw number of productive new editors is highly dependent on the raw number of new editors, this metric is best examined as a proportion. Given that identifying reverts is computationally difficult, the following plots were generated by randomly sampling newly registered users stratified by registration month.

German Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

English Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

Spanish Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

French Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

Polish Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

Portuguese Wikipedia

Productive new editors per new editor. The proportion of productive new editors is plotted by registration month for two values of $t$.

Productive new editors per newly registered user. The proportion of productive new editors is plotted by registration month for two values of $t$.

Factor comparison of n and t

Factor of n. The factor of difference between proportions of productive new editors for different values of $n$ is plotted.

Factor of t. The factor of difference between proportions of productive new editors for different values of $t$ is plotted.

Usage

User interface experiments at the Wikimedia Foundation apply the productive new editor metric across an entire cohort to generate the productive newcomer proportion. For example, in an A/B test, we might compare the proportion of productive new editors in a control group to a test group. This allows us to have a very basic understanding of whether the A/B test led to more new editors making productive contributions.

See the following research reports for examples of this type of usage:

• Article feedback/Stage 3/Conversion and newcomer quality
• VisualEditor's effect on newly registered_editors/Results
• Onboarding new Wikipedians/OB6