Research:Privacy-conscious AB testing at Wikimedia Foundation
Every time we need to segment Wikimedia project users we end up having to write code on varnish and discuss whether we are segmenting per IP, per device or per “user”. When we want to launch a feature that might be disruptive, we normally follow the path of enabling it on a small wiki and extrapolating those results to a bigger wiki, which does not always work.
The goal of this document is to explore the design of a basic A/B testing framework with strong privacy constrain such we do not end up having to modify varnish code every time we want to launch a specific feature. Our initial objective is to provide a simple framework to obtain data statistically valid in the absence of logged-in users or sessions. Thus, in the initial design, we are only considering readers using the Wikimedia sites anonymously. We do not consider editors.
Timeline
[edit]Design Document to be delivered by March 2017
References
[edit]- Designing and Deploying Online Field Experiments, 2014: https://hci.stanford.edu/publications/2014/planout/planout-www2014.pdf
- PlanOut: https://facebook.github.io/planout/
- Overlapping Experiment Infrastructure: More, Better, Faster Experimentation: https://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36500.pdf
- Seven Pitfalls to Avoid when Running Controlled Experiments on the Web: https://ai.stanford.edu/~ronnyk/2009-ExPpitfalls.pdf