Research:Onboarding new Wikipedians/OB5
 | This page in a nutshell: We observed that the test cohort is significantly more successful when looking at the 1+ article edits metric, among others, but still did not produce a statistically significant increase in active editors (5+) compared to control. This compares favorably with previous versions of the onboarding experiment series. |
In this iteration of onboarding experiments, we will compare a control group to one delivered a new landing GettingStarted page, as well as a new navigation bar delivered to those who accept GettingStarted tasks, which lets them skip to a new random task, go back to the GettingStarted landing page, or generate a short guided tour showing them how to complete the task.
-
The landing page in this experiment -
Task article with navigation bar
Overview
[edit]Cohorts
[edit]
- Control - delivered no GettingStarted
- Test - delivered GettingStarted, which breaks down in the following funnels:
- gettingstarted-copyediting - editors who chose the copyediting task
- gettingstarted-clarify - editors who chose the clarification task
- gettingstarted-addlinks - editors who chose the add wikilinks task
- returnto - editors who opted to click the "No thanks, return to the page I was reading" link to their internal referrer
Metrics
[edit]- 1+ NS0 edits in 24 hrs
- 5+ NS0 edits in 24 hrs
- Revert rate
Research questions
[edit]- How do the conversion rates of gettingstarted users exposed to this version of the landing page compared to the control group? Specifically...
- Does the navigation bar's functions for returning to Special:GettingStarted or skipping to a new random article enhance conversion rates?
- Does the guided tour generated by "Show help" enhance conversion rates and editor productivity?
- How do these relative conversion rates compare to the results OB2 and OB4?
- How do the conversion rates among task types (copyediting, clarification, add links) compare to each other?
- What is the quality of edits made in each task funnel, as measured by revert rate and hand-coding?
- Is there a difference (and what kind) in the activity of editors who authenticated an email address and those who did not, in both control and test? Note: Email is confound in this experiment, as it is likely that users who do authenticate an email address have naturally higher levels of engagement, in addition to receiving gettingstarted notifications.
Data
[edit]We ran the test for one week and logged activity for users in the two groups via the Schema:GettingStartedNavbar log. We introduced a fix on May 20 due to a bug that affected logging of users clicking on the ReturnTo button in the control condition.
| Schema revid | start | end | log version | notes |
|---|---|---|---|---|
| 5467309 | 2013-05-16T21:57:59Z | 2013-05-20T23:37:37Z | 1 | Missing users in the returnto funnel |
| 5496876 | 2013-05-20T23:38:26Z | 2013-05-23T20:31:09Z | 2 | Fixed logging |
Cohort analysis
[edit]We defined the following cohorts in order to measure their respective productivity. We included in the sample all new registered users on the English Wikipedia (excluding mobile registrations) during the two rounds of the split test (log version: 1, 2). Cohorts in the GettingStarted funnel were obtained by checking for the first click upon account creation. Cohorts in the ReturnTo funnel were obtained by checking for impressions on an editable article linked via the ReturnTo button.
| Cohort ID | Unique users | Description | % of total bucketed in group |
|---|---|---|---|
| All users | |||
| e3_ob5a | 12,201 | Users in the control group | 100% |
| e3_ob5b | 12,375 | Users in the test group | 100% |
| GettingStarted funnel | |||
| e3_ob5b_gettingstarted-any_page-impression | 4,047 | Unique users in the test group who clicked on one of the three GettingStarted buttons upon successful account creation. | 32.7% |
| e3_ob5b_gettingstarted-addlinks_page-impression | 1,446 | Users in the test group who clicked "Add links" upon successful account creation. | 11.7% |
| e3_ob5b_gettingstarted-clarify_page-impression | 981 | Users in the test group who clicked "Clarify" upon successful account creation. | 7.9% |
| e3_ob5b_gettingstarted-copyedit_page-impression | 1,661 | Users in the test group who clicked "Copyedit" upon successful account creation. | 13.4% |
| ReturnTo funnel | |||
| e3_ob5a_returnto_page-impression | 693** | Users in the control group who returned to an editable page via the ReturnTo button upon successful account creation. | 5.7% |
| e3_ob5b_returnto_page-impression | 1,076 | Users in the test group who returned to an editable page via the ReturnTo button upon successful account creation. | 8.7% |
Notes
- Due to a bug that affected the logging of events in the control condition in v1, the e3_ob5a_returnto_page-impression cohort is incomplete and its size cannot be used for historical comparison.
Live account rate
[edit]We measured the proportion of "live accounts" or users in each cohort clicking at least once on the edit button on an article in the main namespace within 24 hours of registration (measurements taken 24 hours after the last valid registration).
| ID | Unique users | Live accounts | % |
|---|---|---|---|
| e3_ob5a | 12,201 | 4,142 | 33.9% |
| e3_ob5b | 12,375 | 4,801 | 38.8% |
We found a significant 4.9% difference in the proportion of live accounts between test and control [X² = 62.16, N = 24,576, p < .01]. The live account rates for the other cohorts are the following:
| ID | Unique users | Live accounts | % |
|---|---|---|---|
| GettingStarted funnel | |||
| e3_ob5b_gettingstarted-any_page-impression | 4,047 | 1,731 | 42.8% |
| e3_ob5b_gettingstarted-addlinks_page-impression | 1,446 | 521 | 36.0% |
| e3_ob5b_gettingstarted-clarify_page-impression | 981 | 494 | 50.3% |
| e3_ob5b_gettingstarted-copyedit_page-impression | 1,661 | 728 | 43.8% |
| ReturnTo funnel | |||
| e3_ob5a_returnto_page-impression | 693 | 320 | 46.2% |
| e3_ob5b_returnto_page-impression | 1076 | 450 | 41.8% |
1 edit in 24 hours
[edit]We measured the proportion of users in each cohort completing their first main-namespace edit within 24 hours of registration (measurements taken 24 hours after the last valid registration).
| ID | Unique users | Editing users | % |
|---|---|---|---|
| e3_ob5a | 12,201 | 2,831 | 23.2% |
| e3_ob5b | 12,375 | 3,081 | 24.9% |
We compared the proportion of threshold-hitting users in each group and found a significant 1.7% difference between test and control [X² = 9.55, N = 24,576, p < .01]. The threshold-hitter rates for the other cohorts are the following:
| ID | Unique users | Editing users | % |
|---|---|---|---|
| GettingStarted funnel | |||
| e3_ob5b_gettingstarted-any_page-impression | 4,047 | 928 | 22.9% |
| e3_ob5b_gettingstarted-addlinks_page-impression | 1,446 | 230 | 15.9% |
| e3_ob5b_gettingstarted-clarify_page-impression | 981 | 294 | 30.0% |
| e3_ob5b_gettingstarted-copyedit_page-impression | 1,661 | 409 | 24.6% |
| ReturnTo funnel | |||
| e3_ob5a_returnto_page-impression | 693 | 201 | 29.0% |
| e3_ob5b_returnto_page-impression | 1,076 | 301 | 28.0% |
5+ edits in 24 hours
[edit]We also measured the proportion of users in each cohort completing 5 or more main-namespace edits within 24 hours of registration (measurements taken 24 hours after the last valid registration). The proportion of 5+ threshold hitters varies insignificantly between the two conditions.
| ID | Unique users | Editing users | % |
|---|---|---|---|
| e3_ob5a | 12,201 | 468 | 3.8% |
| e3_ob5b | 12,375 | 489 | 3.9% |
The 5+ threshold-hitter rates for the other cohorts are the following:
| ID | Unique users | Editing users | % |
|---|---|---|---|
| GettingStarted funnel | |||
| e3_ob5b_gettingstarted-any_page-impression | 4,047 | 142 | 3.5% |
| e3_ob5b_gettingstarted-addlinks_page-impression | 1,446 | 39 | 2.7% |
| e3_ob5b_gettingstarted-clarify_page-impression | 981 | 40 | 4.0% |
| e3_ob5b_gettingstarted-copyedit_page-impression | 1661 | 65 | 3.9% |
| ReturnTo funnel | |||
| e3_ob5a_returnto_page-impression | 693 | 29 | 4.2% |
| e3_ob5b_returnto_page-impression | 1,076 | 45 | 4.2% |
Revert rate
[edit]| ID | Unique users | Reverts | % |
|---|---|---|---|
| e3_ob5a | 12,201 | 998 | 8.1% |
| e3_ob5b | 12,375 | 1,046 | 8.4% |
Funnel analysis
[edit]Guided Tour
[edit]The following is the breakdown of users in the e3_ob5b_gettingstarted-any_page-impression group taking the Guided Tour embedded in the navigation bar within 24 hours of the end of the split test.
| ID | Total users | Step 1 | Step 2 | Step 3 | Step 4 | Step 5 |
|---|---|---|---|---|---|---|
| e3_ob5b_gettingstarted-any_page-impression | 4,047 (100%) | 366 (9.0% / 9.0%) | 322 (88.0% / 7.9%) | 274 (85.0% / 6.8%) | 173 (63.1% / 4.3%) | 54 (31.2% / 1.3%) |
Conclusions
[edit]Though we have not yet made a gain in the rate of active editors hitting the 5+ edits threshold, we have so far observed the largest gain in new editors hitting the 1+ edits threshold of any of our controlled tests of Getting Started. Funnel analysis and further analysis of subcohorts (such as by task type and those who opted in to the guided tour) is merited, to understand more detail about which features may be driving this effect.