Wikimedia monthly activities meetings/Quarterly reviews/Growth/December 2013

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Growth team on December 4, 2013, 1:30pm–3:30pm

Present: Howie Fung, Steven Walling, Erik Möller, Tilman Bayer (taking minutes), Terry Chay (taking a lot of minutes, too), Sue Gardner

Participating remotely: Aaron Halfaker, Matt Flaschen & Pau Giner

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Slides

Agenda

Review Annual Plan target and 2013-14 goals
What we've shipped
What we've learned about our users
What's next on the roadmap
Discussion about team's progress and needs

Annual Plan Goals

mw:Growth/2013-14 Goals

recall our annual goal: 2400 additional active eds (5+) across all languages
was derived from projections of 3rd iteration of GettingStarted (GS)
would stop decline (flat), if not grow numbers

acquisition:
- get more anon editors to create accounts
activation:
- onboarding - tested successfully on enwiki, but need to roll out broadly
- improve article creation
retention and reactivation
- provide compelling task suggesitons and workflows
- fix reverts and audit bots

key user stories (3 people)

"Max": anon, signing up
"Adam": made 1 edit (perhaps came in from GS), what's next
"Sarah": wants to start new article, but doesn't know how

What we shipped

6th and final onboarding A/B test

(already talked about results in metrics meeting:)
6th test was about sending people to landing page (choice of 3 tasks) vs. returning them to the article they were on before signing up

A more complex workflow required more nuanced analysis, but was more rewarding to key editors:
Signups returning to non-editable pages (e.g. non-articles): 30% CTR, 17% edit rate ("find pages")
Signups returning to editable article: 51% CTR, 52% edit ("edit this page"), similar to mobile rates: ~40%
Sue: so this is the best answer we found so far?
Steven: yes. so this successfully addresses people with the motivation to make that kind of edit

API for new editor tasks

- works
- made accessible to others, e.g. Mobile
- also easier to generalize without UI

Erik: API deployed already?
(Steven:) not quite, it is merged and riding the deployment train to enwiki

Matthew: make tasks based on different thing (more clear), use categories, also refactored,
Erik: where is that defined? community-editable / manually config change for every language?
Steven: no, a config variable
Steven: next sprint: refactoring so it refers to Wikidata items instead...
Howie: already implemented the evaluation of user's contribution history that we talked about earlier?
Matthew: not yet
Erik: use for microcontribs? Mobile using this already?
Steven: Yes. Mobile has reviewed patches and tested already.

GuidedTour support

Major refactoring. Now used by community and other teams, too: general purpose tool for influencing user behavior on site

Example tour (from Commons on nominating content for deletion)
Wikipedia Adventure on enwiki - needs work, but they use GT in inventive ways
BetaFeatures

What we've learned

Aaron joined us mid-quarter, a lot of what we learned about users comes from his work

Onboarding

ready to productize

answered some of the key questions:

what kind of users can we target?
how do we get them activated?

Erik: how to get people from 1 to 50, 100 edits?
current version takes you from one task to the next
Steven: that's still how it is
Erik: do I get that when clicking "edit" for this page?
Steven: no, no "keep going" for that
Erik: How successful?
Aaron: We've never answered this question
Erik: do "game mechanics" work for keeping users engaged?
Steven: we do see people who edit several GS articles in a row. but...
Erik: current implementation could do more to make it fun to stay in this process
not instant reponse, still reload full page, just a centralnotice-like note on top
Steven: Desktop and mobile are converging. The flow of signup → send back to article with reminder of what they were doing → none of them have done the "keep going" part yet
Howie: not a lot of work yet on toolbar thing
only hook for getting them back is the echo notification
let's talk about this later in forward-looking part

Anonymous editors

Steven: Research:Anonymous editor acquisition/Volume and impact:

Makes 2 article edits in a 15 min session (vs. 30 min for registered)
historical revert rate 80% retained (not reverted), vs. 90%+ for logged-ins (per http://stats.wikimedia.org/EN/EditsRevertsEN.htm , cf. http://infodisiac.com/blog/2013/07/new-edit-and-revert-stats )/

Types:

first-timers kicking the tires
repeat anon contributors (chose not to register
incognito registered Wikipedians (e.g. privacy reasons for logging out)

vandals could be any of these three
haven't quantified these yet, but important to keep them in mind

Aaron:
Anonymous editing as pre-onboarding
http://meta.wikimedia.org/wiki/Research:Anonymous_editor_acquisition/Volume_and_impact
used checkuser table (IPs + user agents)
correlated that with RC table for logged-in edits
for newly reg users, found anon edits from same IP within 1h prior, those were assumed to be the same session (by same person)
Steven: result:

6.9% of newly registered users had edited anonymously within 1h prior to registration
users with an anon edit session are 6% more likely to be productive (1+ edits not reverted in 48hrs) in their first edit session (after registration).

Previously, we treated all non-registered editors and readers as one black box. Today, we think of them as sub-types even before registration:

readers
anon editors (7% min from http://meta.wikimedia.org/wiki/Research:Anonymous_editor_acquisition)
potential editors, who showed signals of intending to edit (10% min - from http://meta.wikimedia.org/wiki/Research:Account_creation_campaigns/Anonymous_editors )

Conclusions:

Anons are a really valuable group to acquire
validated the idea of anon editing as pipeline to more involvement

Sue:
long tail vs. very active core of contributors
e.g. Aaron Swartz' findings
Matt F: depends on total size of edits or # of edits
Steven: depends of when you look. Aaron Swartz' study was in 2006, larger proportion of anons back then
Howie: not sure... a while ago, I did a scatter plot number of edits vs. bytes added
Aaron: "creating, destroying and restoring value" paper (2007) found that 0.1% of Wikipedians add 40% of viewed words
Sue: the hypothetical idea of forcing all anons to register so that they can enjoy the benefits of registration, would ignore this
we don't capture these people (anons) in our active editors numbers today
Steven: ...

Wikipedia Article Creation

learned some quantitative and some qualitative things:
Research:Wikipedia article creation
differences in workflows between language editions
AfC (Articles for Creation) process on enwiki
Erik: suspect AfC success ratio (probability of not being deleted) is lower than for redlink article creation

Aaron: File:Allwiki.survival_proportion.articles.lm.svg
Unsurprisingly, articles by oldtimers (right) are mostly kept, that rate has been increasing over time, new users (day, week, month) see general decrease over time
Howie: what about autopatrolled new articles? Erik: shouldn't matter here
(Aaron:) autocreate: these are usually experienced users from other wikis
anon success rate higher than that of newcomers, suggests that a lot of anons are experienced editors
Erik: could we run this on other wikis too?
Aaron: will have data on dewiki and all other large wikis soon

File:Monthly_surviving_prop.lm.articles.svg (enwiki)
3 newcomer classes (day, week month)
survive ratio is actually increasing on en

Sue: this is surprising, suspect reason is on the supply side
Aaron: caveat: this counts namespace 0 page creations
AfC not picked up
Howie: so moves don't show up
Sue: numbers don't make sense
Steven: queue of unreviewed articles in AfC could explain it
Sue: but is the volume of AfC sufficient to explain it? Steven: perhaps yes, will find out
Matthew: e.g. a borderline non-notable company article, earlier would have been deleted, now more likely to be overlooked
Sue: so is AfC doing more harm than good? is it capturing the sludge that would have had to be deleted otherwise, or is it capturing sludge *and* holding up good contributions?
Steven: enwiki community pushes us to create a draft namespace, we feel data is not sufficient yet Research:Wikipedia_article_creation
AfC is kind of a draft namespace already, but broken. community wants a non-broken version of that. but that needs solid testing

Newcomer Survival Models

Steven:
http://meta.wikimedia.org/wiki/Research:Newcomer_survival_models
predict whether newcomer will stick around, and find out what factors predict that
time spent editing is a good predictor
number of sessions: after three sessions (regardless of number of edits), only 29.3% of newcomers will drop off
Aaron: "how to get people to 5 edits" really means "how to get people to stick around"
I think edit sessions is a more meaningful measure
it might be that they just have more intrinsically motivated
but it could also be possible that measures to keep newbies around for longer could have a great effect on retention
Howie: How does it work with Dario's standardization process?
Aaron: This is more background work. Will be proposing to Dario that survival threshold is the 3 edit sessions.
Howie: we have metrics that we use at the level of WMF in general. this might be more appropriate as a team-level metric. otherwise might become inundated with too many metrics
Aaron: I agree
( looking for inflection point at team level: the point at which it becomes exponentially more likely)

What's next on the roadmap

Steven:
launch API soon, then shepherding API, make sure it works
Launch onboarding with the version we have decided on
on e.g. dewiki, frwiki, hewiki
next, start anon acquisition tests
also important: article creation (AfC/draft namespace). but this is thorny, lots of work
team is still small, will grow by one, perhaps two in next few months
Erik: enwiki draft namespace will happen
Steven, Matthew: open question: new draft space replacing AfC, vs. launching alongside it
Steven: Pau is working on designs currently

Discussion about team's progress and needs

Steven:
1. How's our progress?
Areas we've talked about (onboarding, anons, article creation) are required complements to other editor engagement work, for turning things around.
don't have the manpower yet

2. What does the team need?
A) get some irons out of the fire (one project at a time)
right now, three different projects. also need to do e.g. maintenance on account creation
Howie: team needs to change some internal, processy things to get projects through pipeline more quickly
within the team, be more ruthless about prioritizing. also, handle external pressure
Steven: yes, e.g. community RfCs and how they match team's roadmap
Sue: internal pressures from within WMF? no
Steven: some stages don't require whole team, that's one reason we have several projects going at once
Matthew: break down cards into tasks that ares small enough to be manageable
Howie: are you doing retrospectives?
Steven: no
regarding external pressure, community liaison support could free up a lot of Steven's time as PM
Erik: but manage expectations about liaison's ability to set specifications/requirements
alternatively, could do interviews, research
Howie: in previous reviews, had conversation about team members having to do outside work (e.g. Ori). Is this still an issue?
Steven: not really. everybody putting in 100% (save some really small and worthwhile work)
Erik: work with e.g. Mobile (API) is precisely within team's mandate

B) unclog the hiring pipeline
candidate quality at beginning of pipeline is much better than e.g. a year ago
but inbetween, some friction
long lag between acceptance by everyone (even including Sue), and candidate receiving package. even though all other departments (Legal, HR, ..) did their best to do it fast
Sue: I don't think three weeks is that bad in general
Erik: in general, we won't hire as fast as a startup. we are small but have high standards
that said, parts of process are needlessly slow:

candidates handed around inefficiently between hiring managers
conflation of HR and Legal. international legal stuff happens only after HR is already done...

Sue: yes, single-tracking instead of multi-tracking
Steven: should get all stakeholders together and map out the whole process
Erik: what we are doing about it currently:

Legal will work on standardizing contracts (they have Andrei now), hiring from outside US will be more routine
install panel-based interview process, instead of several interviews with different teams

these two things should make a huge difference
Steven: things are better than they used to be, individual aspects improve, it just feels to me that the whole thing does not have ownership
Erik: also, better application tracking system (ATS)
Sue: could flag urgent cases to Joady, especially regarding reference checks
in general, 2-3 weeks might be fine for most cases
Erik: hiring managers or their bosses should be able to help expedite process
Sue: yes, but also flagging, as a separate measure

C) adapt our practices to being a mostly remote team
Steven: I'm the only person on the team who works in SF
Pau: experience from language engineering team, which has the advantage of having close timezones
also, we have retrospectives (as Howie said), which helps to structure things
and it's a small team
Howie: getting feedback from other designers on your Growth work?
Pau: yes, sharing things among designers using the weekly design meeting
Matthew: we are doing pretty good, but retrospectives could help. We should keep in mind that with travel there is recovery time.

Erik: I think you guys are doing well, especially as you lost two very productive engineers, and have to deal with contentious community conversations. Props to Steven as a PM who addresses such issues directly as they occur.

D) consider working with Mobile more closely

Sue: would like to talk about AfC/Drafts. I have strong opinions, which might be wrong and misguided though ;)

(break, followup discussion about AfC with some participants)