Research talk:Notifications/Experiment 1

How do we handle web vs email?

• we won't be able to A/B test the delivery method per se so we should focus on making each condition internally consistent --DarTar (signed by EpochFail))

What happens at the end of the test for users in a condition other than the current default?

• Reverting user preferences is likely to be a disruptive/frustrating experience for newcomers. --EpochFail (talk) 22:00, 29 May 2013 (UTC)
• Leaving preferences permanently set to the conditions used in the test (unless users re-opt in or out) will allow us to measure the longer-term effects of Echo (particularly after a short, 1-week test) --DarTar (signed by EpochFail))

Will users have the ability to change their prefs during the test?

• Removing the ability to change prefs is going to be very disruptive and require extra engineering effort. We can track pref changes during the experiment --EpochFail (talk) 22:00, 29 May 2013 (UTC)

Should users still receive the legacy talk page notification?

• This would help us answer the question, "What does Echo add?", but would not help us understand how notifications affect user behavior, generally. I think this is a fine so long as it's tractable to re-enable the legacy notification for a subset of users. --EpochFail (talk) 22:00, 29 May 2013 (UTC)
  • The legacy talk page notification is enabled by default when Echo is disabled, so that comes for free. Whether to display it or not in the No-Echo condition is a question for the PM. --DarTar (talk) 17:23, 30 May 2013 (UTC)

What's the rationale for a 2 week experiment?

• We are going to need as many users as possible per experimental condition if we want to infer anything about preference changes --EpochFail (talk) 22:00, 29 May 2013 (UTC)
  • How many is as many users as possible --DarTar (signed by EpochFail))
    • Given my preliminary work, we'll need to be able to determine significant differences in proportions that land around 0.005 for the preferences that are most rarely changed (e.g. opting into emails about Article Links). In order to perform the standard proportion tests, we'll need to capture at least 25 preference changes per preference. 25/0.005 = 5000 which is a little less than 1/3rd of the number of newly registered users who edit that we get in two weeks. --EpochFail (talk) 22:00, 29 May 2013 (UTC)
• Also, we have a two week time window in June before Dario flies away. --EpochFail (talk) 22:00, 29 May 2013 (UTC)
  • Cons: we also have VE tests coming up, likely around mid June --DarTar (signed by EpochFail))