Research talk:Newcomer task suggestions/Work log/2014-09-24

Wednesday, Sept. 24th

I ran the usual barrage of tests for the users in the experimental conditions.

Productive revisions (24h). The geometric mean number of productive revisions that editors saved in the first 24 hours is plotted by wiki and experimental condition.

Productive editors (24h). The proportion of editors who save at least one productive edits in the first 24 hours is plotted by experimental condition.

Productive editors (2 productive edits, 24h). The proportion of editors who save at least two productive edits in the first 24 hours is plotted by experimental condition.

Activated new editors (24h). The proportion of activate (5+ edits) editors in the first 24 hours is plotted by experimental condition.

Article revisions (24h). The geometric mean number of article edits saved in the first 24 hours is plotted by experimental condition.

So, here's the breakdown:

• I thought that #Productive revisions (24h) would be most likely to show a difference if there was something good happening. Here, we see no significant difference from the control on any wikis we observed.
• If Task Recommendations (TR) is not helping increase the number of productive edits that newcomers make, maybe it's increasing the proportion of editors who make at least one. If so, #Productive editors (24h) would show a difference. It doesn't. But maybe that's because recommendations don't come until at least one article edit has been made. So I checked whether setting the threshold at two productive edits would see a difference. Yet #Productive editors (24h) is the same story.
• OK. Maybe TR is helping newcomers make more edits to articles, but they are struggling to get them to stick. In which case, we'd see significant differences in #Article revisions (24h), but we don't.
• The only place where we see a significant difference from the control is nlwiki, where the control performs marginally better than the conditions that have the flyout in #Activated new editors (24h). I think that is statistical noise.

Preliminary thoughts

Effect (if any) is too small to be detected at the very high number of observations that we have. This might make us sad, but it also doesn't look like we're causing any harm. I propose that we (1) look again when we have two weeks of data and (2) look at retention stats when have waited long enough to see if newcomers will be retained. --Halfak (WMF) (talk) 01:04, 25 September 2014 (UTC)