Research:Notifications
Key Personnel
[edit]- Aaron Halfaker
- Dario Taraborelli
- Fabrice Florin
- Oliver Keyes
Project Summary
[edit]The aim of this project is to evaluate the usage and impact of the new notification infrastructure (also known as Echo) designed by the Wikimedia Foundation and first deployed on the English Wikipedia on May 1, 2013.
Research Questions
[edit]Our primary focus is on two sets of questions:
- Usage
- Who is sending and receiving notifications?
- What is the volume of notifications delivered by type?
- By comparison, the total number of mostly-MediaWiki email notifications sent is about 3.3 billions per year.[1]
- How many notifications are delivered on-site vs other delivery methods?
- What is the relative volume of notifications triggered by bots?
- What is the conversion rate for different types of notification?
- How does the use of private notifications compare to public ones (e.g. talk pages)?
- Impact
- How do notifications change user behavior?
- Do notifications drive new editor retention and productivity?
- Are there notifications that increase the impact of negative warnings?
- At what rate of emailing do user's react (turning notifications to web only or opting out entirely)?
Data
[edit]Data for research on Echo is mostly obtained by instrumenting MediaWiki via EventLogging. We are using three sources of data, described by the following models:
- Schema:Echo
- logs notifications generated by the system and their corresponding delivery method. Notifications logged with the 'email' delivery method indicate that they are staged for delivery to users with validated email addresses and not opted out, not that they have effectively been delivered.
- Schema:EchoMail
- logs how many email notifications are effectively being sent by MediaWiki, categorized by type (individual, bundled, digest)
- Schema:EchoInteraction
- logs user interactions with individual notifications.
We will use this data to measure conversions and other funnel metrics. We will also use cohort analysis (via the UserMetrics API) to run controlled tests and compare how the productivity and retention of different groups of users is affected by notifications.
On top of EventLogging data, we will monitor changes in preferences (user_properties) to determine the timing and volume of opt-outs or changes in default delivery methods.
We also rely on usertesting sessions and short surveys to study usabilty and UX design of this feature.
Controlled tests
[edit]Privacy and licensing terms
[edit]Unless otherwise specified, all of the above data is private and subject to the Wikimedia Privacy Policy. Daily aggregates are released in the public domain and can be obtained via the Editor Engagement dashboards.
External links
[edit]- Echo on mediawiki.org
- Echo FAQ on the English Wikipedia
- Real-time notification dashboards