Research:Mobile editor engagement/Left navbar signup CTA test

The left navbar signup CTA test is a controlled experiment designed to test if displaying an edit call-to-action to newly registered users signing up on a mobile device increases their edit completion and activation rate. The test will be initially enabled for Beta users only. The actual controlled experiment will be eventually rolled out for users of the stable version of the mobile site.

Timeline

Test starts (stable)

deployed to en.wiki on Thursday, February 06 at 11am PST

Test ends (stable)

Experimental design

This is the Guider that is shown to Mobile users on enwiki after they sign up from the Left Nav Bar

• We will select a sample of new users signing up via a mobile device on the English Wikipedia for a period of at least two weeks. Only users registering via the Signup link in the left navigation bar are considered eligible for this test.
• We randomly assign eligible users in the sample to one of the following two conditions, based on their user id

Tutorial (test)

after signing up and being redirected to the referring page, users see a call to action inviting them to edit, if the page is editable (and in main namespace).

No tutorial (control)

users are redirected to the referring page.

• Users remain bucketed in the experimental condition for the duration of the test (initially set to 2 weeks). At the end of the test, they reverted to default condition.

Research questions

• Does displaying a CTA increase the edit completion rate of new mobile registered users (proportion of users registering via the left navbar CTA funnel and successfully editing the page they came from)?
• Does the tutorial affect the overall activation rate of these users?
• Are there differences in the quality of contributions made by users in each condition, as measured via unreverted edits?

Notes

• The CTA is displayed only on pages on the main namespace.
• We will disable the CTA on non-editable pages. However, we will explicitly log bucketing/landing page impression events on these pages for data QA purposes via the isEditable field.
• We do not capture client-side interactions with the CTA (such as clicks on the accept or cancel button). The source of an edit screen impression is logged via the clickSource field in the schema.
• Users won't be able to create new pages: only activity on existing pages will be logged.
• During the test, post-edit feedback will be displayed to all users in Beta but only after the completion of the first edit, which won't affect the validity of the test to measure first-edit completion rate.
• Client-side edit activity for the purpose of the test is only logged on the immediate referral page. We won't store client-side data on other pages the user visits after registration. This includes the page itself if for some reason the user returns to it after browsing away. Further edits will be logged via revtags as usual.

Data sources

For this test we will use a combination of data from the MediaWiki database and EventLogging logs (Schema:MobileWebEditing and Schema:MobileLeftNavbarEditCTA).