Research:New editors' first session and retention

WSoR 2011

Contact

Aaron Halfaker

University of Minnesota

---

Research:Projects

This page documents a completed research project.

In this sprint, I'll be looking at how editors behave in their first session editing the encyclopedia, how other editors react to their work and how that predicts retention.

Looking at the first session of activity is important because it represents the first interaction an editor has with the community under their registered persona. I hypothesize two things:

• The raw amount of activity that an editor engages in during their first session is a useful proxy for their initial investment/motivation to contribute.
• Editors are most vulnerable to rejection at this crucial time.

Previous work shows that editors are most negatively affected by being reverted when they are new to the encyclopedia, but I intend the above hypothesis to assert that declines in the survival of new editors in Wikipedia can be tracked by rejection in this specific timespan.

Definitions

• Edit session: An edit session is a single span of work performed by an editor without leaving Wikipedia to do something else. I intend to capture what an editor does while they are still "present".
  • I quantify a session as a sequence of edits across the encyclopedia where the associated timestamps differ by less than an hour. It's common practice for user sessions on websites with sensitive information (banks, shopping carts, etc) to use 1/2 hour to one hour as a timeout. This is because users will generally take less than an hour to perform the next action if they are still presently at their computer. This should be a good proxy.
• Early survival: Survival is a difficult metric to measure since it is a boolean (0,1) value attributed to a non-boolean phenomenon. It is intended that for whatever underlying approach, survival should be a measure of crossing some important threshold. Any arbitrarily chosen threshold will fail to convey the nuance of survival, but identifying difference between groups of editors crossing or not crossing even arbitrary thresholds can be informative.
  • I define survival as making at least one edit more than one month after the first edit session. I assume that most everyone who comes back for at least one month to edit has surpassed whatever important initial barriers there are in the system/community.
  • Since there is a sunset to the data (data on each editor ends in July 2011 because that is when I ran the analysis) I cut the data off 6 months before now and introduced a 6 month artificial sunset towards the definition of survival. This means that I only consider an editor "surviving" if they edit at least 30 days after their first edit session, but not if the next edit comes more than 6 months away. This modification doesn't substantially affect my results.
• Early rejection: The amount of an editor's work that is rejected by the community is likely to have an effect on an editor's motivation to continue doing work. Measuring the proportion of an editor's first edits that were reverted or deleted should be a good way to understand how a new editor perceives community reaction to their work.
  • I define early rejection as the proportion of an editor's edits that are reverted or part of a page that has been deleted. Since editors with few initial edits will tend to have 0% or 100% of them rejected, they introduce a lot of noise into the measure so I include the first three edit sessions worth of edits and limit my analysis to editors who have at least 4 edits to judge the rejection rate properly.

Methods

Number of sampled users by first edit year

first edit year	n
2001	690
2002	1,899
2003	8,734
2004	9,994
2005	9,999
2006	10,000
2007	10,000
2008	10,000
2009	10,000
2010	10,000

I randomly sampled up to 10k new editors from each year of Wikipedia history based on that editor's first edit. I specify "up to 10k" because the first few years of Wikipedia have < 10k new editors in total!

For each sampled editor, I followed their initial contributions to the encyclopedia to construct edit sessions (as defined above) to identify what work each editor performed in the first three sittings and captured the necessary metrics to identify "early survival" (as defined above).

Below I present visualizations of the data and report what trends the data suggests.

Results and discussion

How does early rejection affect early survival?

To determine whether early rejection is a predictor of early survival, independent of other factors, I performed a logistic regression over early survival as predicted by the number of edits an editor performs in their first session, the year in which they registered and their early rejection.

Deviance Residuals:
    Min       1Q   Median       3Q      Max
-4.4728  -1.1104   0.7246   1.0146   1.6701

Coefficients:
                          Estimate Std. Error z value Pr(>|z|)
(Intercept)                0.19363    0.01601  12.093  < 2e-16 ***
investment                 1.56204    0.14122  11.061  < 2e-16 ***
year                      -0.27590    0.01642 -16.798  < 2e-16 ***
rejection                 -0.55456    0.01689 -32.825  < 2e-16 ***
investment:year            0.16206    0.14433   1.123   0.2615
investment:rejection      -0.55797    0.12950  -4.309 1.64e-05 ***
year:rejection             0.11042    0.01744   6.330 2.45e-10 ***
investment:year:rejection  0.36084    0.14517   2.486   0.0129 *
---
Signif. codes:  0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 26494  on 19221  degrees of freedom
Residual deviance: 24378  on 19214  degrees of freedom
AIC: 24394

Number of Fisher Scoring iterations: 6

		survival			rejection
year	sample	surviving	prop	se	mean(prop)	se
2002	1899	250	0.132	0.008	0.080	0.005
2003	8733	1112	0.127	0.004	0.110	0.003
2004	9993	1317	0.132	0.003	0.168	0.003
2005	9999	917	0.092	0.003	0.293	0.004
2006	10000	610	0.061	0.002	0.383	0.005
2007	10000	286	0.029	0.002	0.409	0.005
2008	10000	302	0.030	0.002	0.376	0.005
2009	10000	284	0.028	0.002	0.348	0.005
2010	9999	270	0.027	0.002	0.326	0.004

Initial rejection rate for the first edit sessions and survival of editors based on when they made their first edit.

Initial rejection rate and survival by when editors made their first edit.

The above regression suggests a few interesting trends:

1. The number of revisions an editor performs in their first edit session (es_0_edits) is most strongly correlated (1.29) with early survival.
   • This supports the hypothesis that the amount of work an editor does in their first session is a good proxy for their investment in the system.
   • The lack of significance in effect for the intersection of es_0_edits and years_since_2001 suggests that this effect has been a consistent predictor throughout the history of the encyclopedia.
2. The strong, significant effect of time (years_since_2001) suggests that there are other effects that correlate with time that affect retention.
   • This term is helpful in separating temporal effects from the effects of non-time-based metrics like initial_rejection and es_0_edits.
3. Initial rejection is a powerful predictor of early survival (-0.55).
   • The fact that the coefficients for both es_0_edits and initial_rejection are larger than years_since_2001 suggests that they both explain more about the variation in early_survival than any other things that have been consistently changing over time.
   • This is evidence that initial_rejection is a better predictor of early_survival than anything that's been changing over time in Wikipedia.
4. The intersection of the proportion of an editors early work that was rejected and the amount of work that they performed in their first edit session (initial_rejection:es_0_edits) is a significant, negative predictor of survival.
   • This result suggests that editors who initially perform more work suffer more from having their work rejected than editors who perform less work. This fits with psychology about early investment and rejection^{[citation needed]} that would suggest that the more invested an individual is, the more they suffer from rejection.
5. The intersection of the time and initial rejection (years_since_2001:initial_rejection) is significantly positive which suggests that the negative effects of initial rejection have been decreasing over time.
   • This is surprising since I hypothesized that early adopters would have a "thicker skin" for rejection, but the effects appear to suggest the opposite.
6. The significant, positive coefficient for the intersection of all three independent variables (es_0_edits:years_since_2001:initial_rejection) suggests that the negative effect on heavily invested editors has been ameliorated over time.

What are editors doing in their first session?

Controlled proportions of editors entering Wikipedia by initial investment (first session edits) over time.

The Controlled proportions of editors entering Wikipedia plot shows how the number of edits that editors perform in their first session has been changing over time. The y axis has been controlled such that 1 = max(proportion) that group represented in the years since 2001. This allows the change in proportion to be more easily visualized for higher edit count groups. The increasing proportion of editors that perform ~ 1 or 2 edits in their first session and decreasing proportion that perform ~8 or 16 suggests that editors are expressing a lower initial investment in edit than they used to.

Rejection of initial work for editors entering Wikipedia by initial investment (first session edits) over time.

Editors appear to be uniformly more likely to be rejected in their initial activity in the encyclopedia regardless of their initial investment. The Rejection of initial work for editors entering Wikipedia plot visualizes changes in the rejection proportion over time for different editor groups. The consistent trend regardless of the number of edits in the first session suggests that all editors are getting rejected more in their first editing activities recently.

Summary

This work presents 2 crucial findings:

1. Initial rejection is a powerful predictor of retention.
2. Initial investment is a powerful predictor of retention.
   1. New editors are showing less initial investment now than they used to.
   2. The more initial investment, the more negative the effect of rejection.

Future work

• Is the increase in rejection due to curmudgeonly Wikipedians (easy fix) or some inherent switch from content construction (pre-2007) to content preservation (harder fix)?