Wikimedia monthly activities meetings/Quarterly reviews/Reading and Community Tech, April 2016

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Reading and Community Tech teams, April 12, 9:30 - 11:00 AM PT.

Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Presentation slides from the meeting

April 12, 2016, 9:30am-11am

Present in the office: Katherine, Jon Katz, Danny, Anne, Zhou, Stephen LaPorte, Tilman, Sherah, Nirzar, Josh, Maggie, Geoff, RobLa, Joady, Jaime V; remotely: Adam Baso, bd808, Dan Garry, Dmitry, Kevin, Michael Holloway, Niharika, Ryan Kaldari, Trevor Parscal, Wes, Rita Ho, Volker Eckl

Reading

Jon: Welcome broken into 2 areas: 1) goals and progress 2) how do we build our muscles as a team, build what we should be building?

Slide 1

jk: YoY decrease in readership, actually a little misleading. There were two 1-time drops leading to this result. But actually about flat over 3-year period.

Slide 2 - Objective: Lang switching

Adam: first goal for reading web team was making enhancements to language switching one of the most clicked buttons, thought worth spending time making interaction more fluid goal: roll out changes to beta -- we were able to do this, ship some stuff to prod even

we've taken lang-switching button, used to be on bottom, requiring scrolling, now on top doing some research, anecdotally it's confusing for users used to the old way with button at bottom of article

On the right is the language overlay people now get when they tap the btn Old display was fairly minimalist, just some lines of text Now, structured! See a list of preferred langs, user tapped on DE at one point, also see ES which is the user's keyboard lang

We've applied this in other contexts like search and works pretty well, decide to do here too Still working on sorting routines for the langs in the 'all languages' list

Seems like users get less frustrated with new overlay, engage more. We think it helps users look up synonymous term, because scrolling has decreased while language selection is strong. Tends to drag down tap-through (language selection), but numbers are still good and wethink it's a net good for users

So looking pretty good, tho we're still working on how to expose lang switching button (call to action maybe?)

jk: web team's focus is really on new readers, particularly in emerging markets/communties

Slide 3 Objective: Web Speed

(jk cont from 2) we see that we have a goal of reducing page size sent to improve speed reduce image size (went from retina-grade to another v. high grade) -- reduced image size 60%

also shipping less HTML (some of which wasn't being shown anyway)

This improves cost, but does it make it faster? We prob didn't hit our goal, but made some improvement. Need to measure better.

Q4, will roll out lazy images (and references, probably some other odds & ends) -> reduce size, let page load faster.

Slide 4 - Objective: Web Speed / Mobile Article Size

Shows impact on size (ex: Barack Obama page on mobile.) Images are huge % of total article size. Thx to performance team for their work here.

Question: do we know how many images are seen?

• jk: with lazy loading in beta, based on results expect to see 75% reduction in total image size.. which suggest that 75% are not seen
• jk: also, something like 40% of users never open section on mobile

Slide 5 - Collections Feature removed

jk: Removal of collections feature was popular with readers, but NOT editor community intro'd moderation concerns

Community RFC went out jan, near unanimous vote to remove the feature Made some friends in comm, but ultimately had to remove

Hurt Android team's quarterly goal since dependent on the Gather backend

Learning: need to address comm concerns in timely fashion; also, our notion of 'beta' not necessarily == the community's.

Geoff: What was the biggest pushback?

jk: Prob moderation. There were ways but prob outside the existing system. There was a misconception also that we didn't think the feature through. In fact we saw that traffic was low enough that existing mech's would be fine.

Geoff: My experience is it's always a learning experience working with communtiy in rolling things out.

iOS

Slide 6 - iOS: Objective: Data Layer

Josh M: Our goal this Q was last Q's goal be completed -- to release major new update (5.0.0)

Goal of the update, measure of success was retention. Not use just 1-2x but keep using after 7 days (pretty standard app industry measure)

Big hit, great press, low crash rate, moved retention #s significantly

misses: accessibility (visual impairments in particular), user education/support, analytics Our analytics infra is not designed around apps, so kind of working around what it can support

Good relations with Apple, app was quickly approved

Wes : They (Apple) also discovered they had a problem internally b/c of the app?

Josh: Yes, they needed us and some other apps to address the bug uncovered in iOS 9 (i.e. on their side)

Slide 7 - retention, true “app-y” experience

What was 5.0 all about? -Added feed of main page, most read content from pageviews api -Privacy-friendly reading recommendations -- browsing history not persisted in server profile -Deeplinks -Previous version looked like a browser, this version is much more app-y --Scrolling feed is a common pattern --App-specific things, like working with the OS on search, ...

Slide 8 - Comms work

Thx to comms team Juliet worked with us about how to talk with people about new features, press was v. much on message, responded well

Slide 9 installs, active devices graph

Impact: red line is active device count Blue line new installs Nice pop from the press, apple Active device count has risen significantly and stayed Goal wasn't so much to grow a huge audience as retain the audience we have

KM: what is done about getting it featured in the app store?

Josh: unfortunately they had already selected a different WP app to be feature around the time of the launch JM: A contact at Apple has asked about our future release schedule, so that we can be featured going forward Currently still featured under 3D touch apps

KM: They do great promotions, back to school was a big one last year, would be nice to see about more general purpose promotions

Slide 10 - retention numbers

Geoff: Why is there a downward trend in active devices? JM: Becuase of the recent issue with deeplinks not loading Also, download rate not much higher than previous version

KM: We were comfortable with the idea of being sort of experimental this Q, do you have a plan to increase downloads? Jm: still have a ?? JM: Trying to hire an audience manager as well

How to read these charts: 0-day (download day), then % of people who downloaded still using after 1, 2, ... 7 days Retention is a hard metric to move becuase youre really trying to change someone's habits

GB: Good % by app industry standards? JM: 10-15% is generally good for reference apps, more like 40% if you're a social game People are using Wikipedia every day, fact we're not capturing them is a shortcoming

Prob will not be getting to 40% but we can do better Feed is simplistic, can get better

Android

Slide 11 - Android: Objective: Sync across devices

DB: Strategic pillar was "community of readers": cultivate community of sharing content, make it easier to discover Ultimate goal: fully social collections of articles, can comment on, suggest additions to We knew Gather was already beta feature in mobile web, seemed like a good idea to plug in, use their backend

Goals this quarter were steps toward that larger goal:

• -Sync saved pages so they are available when logged into another device
• -But Gather was shut down
• -Explored other options, but they weren't sustainable
• -We still want to achieve this goal, but it's on hold now
• -Talking about how it should be impolemented in MW, work with community to do

Slide 12 - Android: Reading lists

We did press forward with "reading lists", a generalization of the saved pages feature -User can organize pages in multiple lists with unique names & descriptions, for better integration into the reading experience -Believe it can improve retention even if not synchronized -Build in such a way that it can be plugged into a syncable backend when one becomes available -Also substantially delayed by Gather removal, but should be finalized and in beta next week or two -Then we'll have some updated #s on retention

Km: Earlier it sounded like the challenge wasn't synchronization but community issues? DB: Reason sync was the problem is that Gather was disabled, we lost the backend KM: So there's no inherent issue with synchronization? db: no

KM: How do you see moving forward on Comm of Readers work, what to do next? DB: Exploring features like multiple watchlists, maybe have those be sharable; but will require lots of comm input

JK: We're also engaging with community on a more co-creation level for any further features in this arena

Community Tech

Slide 13 - Community Tech - Objective: Prioritize needs

DH: Did big survey, our job for Jan was to report back with our results on what people think we should do We're planning such a report every few mos. First one at the dev summit, then one at Hackathon/Wikimania, then one in fall

Also have goal of on-wiki discussions start to finish on projects; we've had such discussions on small project but no big ones yet Discussing 3 different approaches to cross-wiki watchlists

Reporting back done, on-wiki discussions starting soon

Slide 14 - Community Tech - Objective: Wish fulfillment - core contributors' productivity

Big goal for q3: complete development and ship:

• 1) migrating dead external links to archives
• -Replace links marked as dead in template to links to Internet Archive*
• 2) pageviews analysis tool

Volunteers are already working on both of these and we don't want to discourage them, we want to work with them

Slide 15 - Community Tech - Migrate dead external links to archives

RK: #1 request on wishlist survey was migrating dead ext links to archives Vol named max was already writing bot to do this We built a logging system for it, hosted on tool labs Multiple bots can interface with it, also has API so bots can see last time it was assessed/fixed and can avoid duplicating work Also lets bot know last article it fixed was so it doesn't lose its place

Also contributed advanced detection code. Initially it relied on volunteer hand-marking dead links by template. We wrote code to allow bots to detect dead links.

Good news: first phase (fixing all marked deadlinks on enwiki) is almost finished. There were (#?), now only 60,000 left. Next phase: Deadlink detection phase

GB: What has been community reaction: RK: enthusiastic. People have been asking for this for a very long time. Other (non-english) wikis actually had solutions for this but enwiki never did.

??: Was there a scale issue on enwiki? RK: Probably, enwiki is the largest and has by far the most references Other issue is that the enwiki community wanted to Do Things Right, have good long-term solution

DH: Tool is being/can be internationalized. But there are concerns about convincing other language communities to use our tool rather than what they're already doing

KM: Is this solution as robust for other langs? Does Internet Archive have the same depth for other langs? RK: We worked with Internet Archive to help them extract all the external links from all the wikis, so they could queue them up and try to archive all of them; should be good coverage across wikis.

GB: What was error rate/case requiring human intervention? RK: We don't have that data (b/c involves human interaction)

DH: Template will show when somebody has manually changed the status

Slide 16 Community Tech - Pageviews Analysis tool

RK: Before, most commonly used tool was stats.grok (offsite), it was notoriously unreliable and people were frequently upset Features were limited, usually just static graphs of pageviews Worked with a volunteer (MusikAnimal) to build a nice new tool on tool labs using API developed by analytics team

Currently getting about 15,000 hits/day We worked on multilang support (was initially english only) And data exporting And various UI/bug fixes

DH: Interestingly, every project that we started, someone was already working on it. It'll probably turn out that most of our projects are collaborations. What people want most are usually things people have been thinking a lot about and at least starting to work on.

Also had some staff issues this Q (down a person).

Slide 17 - Community Tech - other successes and misses

Other projects before wishlist survey:

• Gadgets 2.0
• PageAssessments extension -- basically to provide easy interface for projects to keep track of articles and their importance

Both had blockers and took longer than expected, so we depriortized in favor of comm wishlists projects, but hope to finish next Q

DH: Also working on cross-wiki watchlists; will be exciting Q.

Slide 18 - Reading infrastructure - Security

BD: Two projects to report on:

• 1) AuthManager -- steady and slow progress but has taken a backseat to other time-critical issues
• Goal is a more modern, flexible auth interface

2FA for high-risk accounts Password verfication service so we can move them out of primary db and to more isolated area

We also deployed SessionManager -- change to how we handle cookies/session state Rollout issues took more time than estimated. Hoping to be done w/ AuthManager this Q, now looking like mid-late next Q Also based on that experience, feature-flagging AuthManager so can be disabled in case of deployment issues Currently waiting on security review so we can merge the core code into trunk

Hope to get into the next 1.27 tarball release (otherwise will have to support the old auth code for next 3 years)

Slide 19 - Reading infrastructure - API Engagement

Started work Oct last year, but stalled b/c the discovery project we were assisting with had some hiccups, needed fixes This Q, the data pipeline got completed, allowing us to measure all Reqs to Action API, including data never captured before in pageviews table:

• -params posting w/ API requests
• -which API modules are being used
• -whether internal/tool labs or 3rd party users

Allows analysis re: user-agents (we recommend that every user/bot use unique user-agent) Data flowing into Hadoop as of end of last month

Raw usage of Action API has volume roughly equal to enwiki pageviews KM: Were you expecting this #? BD: We suspected so based on the size of the log files, but didn't really know until we got this data

Slide 20 - Reading infrastructure - Other successes and misses

-Non-free image switch: enwiki found they didn't like fair use images as hero images, we changed code to surface this -Got people one-time snapshots of their Gather collections before removal -Major rewrite of API sandbox, folded it into core --In doing this, extended OOjs.ui with new date/time widget -Invented security feature allowing quick expiration of user sessions. Previously took 48h.

Adam B. taking over as manager as I (BD) transfer to comm tech.

Appendix

Slide 21

titles

Slide 22

titles

Slide 23 - Strategy

JK: Q2 was about strategy, this Q more about execution --Greater reliance on design research, working more closely together, focusing on collabs across the org

Slide 24 - Overall pageviews

Pageviews still flat to negative --Important finding to add clarity: If we only know pageviews, don't know why: disintermediation, move to mobile? ---recently got device #s from analytics and it turns out people view 40% fewer pages on mobile devices, so move to mobile is likely in large part driving the drop

GB? : So this could be consistent with increased # of readers JK: Y

Slide 25 - strategy assignments by platform

JK: Three themes, divided by platform so each team can have focus Not sure which will pay off first or fastest

Improve the encyclopedia experience: new experience: extend the experience, add new use cases new readers: extend to places where access has been an issue

Slide 26 - operationalizing our strategy

JK: How do we define success metrics? (see slide) There's overlap but these are the basic buckets.

Slide 27: Community (cover slide)

JK: Lack of relationship with community is one of the biggest problems we faced (#1 for web team) -Not just can't get approval, but aren't even sure what approval looks like

Slide 28 - Developing Useful Engagement Models

We want to move to the right here. Initial sense was adversarial; mostly stuck in the 'customer' bucket now, moving toward partnership/multiplier model

Slide 29 - Mapping Engagement Success

When/how to talk to community? The more early and the more specific the terms, the better the results

Slide 31 - Progress on Engagement (now)

JK: Significant inroads this Q in moving convos upstream rather than presenting comm with finished products and asking how they like them

e.g.: Language switching co-creating RFCs with community members

JK: Haven't yet shipped anything with this model Nirzar: User pages will be

Slide 32 - Progress on Engagement - Reading / Community Tech (long list)

JK: Our on-wiki consultation methods aren't working on mobile. KM: Who owns these methods? JK: in one case I am referring to idealab. I don't know who owns it now, but it was created awhile ago by Jonathan Morgan and HEather.

• : Might be helpful to contextualize some of these charts (e.g. "Adversary --> Customer") could be upsetting without context

Slide 33 - Understanding our Users (cover slide)

JK: Right away when started reading team we realized we didn't know a lot about our users.

KM: Impressed by depth of thinking that's gone into comm engagment, great to see

Slide 34 - Understanding our Users (list)

JK: Now we know about numbers, starting to do ethnographic reserach (Mexico); usability lab running every other week Working on analytics for mobile usage

Slide 35 - Mexico Reader’s Research

AG: Currently working through the data, figuring out presentation

Slide 36 - Looking ahead

Will be heading to Nigeria (May) and India (June)

Slide 37 - Research team work - depth research

JK: Teasers from research team:

• Asked readers: what are you looking for? (in-depth info, overview, specific fact)
• ~25% come for in-depth learning -- about what we thought

Slide 38 - Research team work - motivation research

JK: Also asked: what is your motivation for coming to WP:

• Top answer: Media (I saw a TV show or read a book) -- most often mobile
  1. 2: work or school -- most often desktop

https://meta.wikimedia.org/wiki/Research:Characterizing_Wikipedia_Reader_Behaviour/S3-English_Large_Scale

Geoff : What's the implication in real terms? -Better ways of thinking about, e.g., which users are getting what they need (quick fact) from Google -How to think about session behavior (what they do while they're here)

Slide 40 - Reading: Q4 Goals: Web, Android

Web: Hovercards feature -- allows mousing over link to preview article. Much better for users on slow connections; don't ahve to wait for full page load. 75% of users don't want full article. So better for them. KM: Will we be tracking hovercard hovers? Can we capture: JK: Y, working closely w/ Analytics to capture. Not exactly pageviews. Tricky b/c some will be accidental. KM: Prob not useful for overarching product decisions but good data nonetheless. JK: Data previosly showed pageviews didn't decline that much w/ previews.

DB: Android project for Q4 is feeds; it's the kind of feature that helps drive retention (as Josh's numbers shown) -- all about providing great first impression to the user. Currently just the front page. Want to be able to show variety of interesting content from the start.

JM: There's no backend for the iOS feed, it's all the OS. Would be nice to incorportae backend. DB: Android team owns the mobile content service; we'll be working to encapsulate some of these features.

KM: Sounds like two goals in there...

JK: Sometimes people wonder why platforms don't all look the same all the time; this is an example where we're testing on one platform then scaling

Slide 41 Reading: Q4 Goals: iOS, Infrastructure, Community Tech

JM: Launched partial support for univeral links --Now support top 6 langs --SWAT deploy is scheduled for thurs to do what remains to be done on universal links

Apple reached out re: handling all searches on the device in standardized way

Out of time.