Research:Collapsed vs uncollapsed section view on mobile web
The mobile web version of Wikimedia projects currently shows all top-level sections below the lead section collapsed (when displaying a page such as https://en.m.wikipedia.org/wiki/Barack_Obama on a phone-size display). Users can tap on a section heading to show the content, and to collapse it again.
To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment in December 2015/January 2016 (setup: phab:T120292), where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed. (For the most part this was not an A/B test in a strict sense, because we are not comparing the same metric for both groups.) It was a followup to phab:T118041, which preceded that experiment.
This project is about the analysis of this data with respect to the main question of how readers use Wikipedia differently in these two settings and which one might be preferable. Some other data from the same instrumentation has also been published as part of Research:Which parts of an article do readers read and of a March 2016 tech talk.
Methods
[edit]We collected data with an EventLogging instrumentation detailed at Schema:MobileWebSectionUsage.
Policy, Ethics and Human Subjects Research
[edit]Data was collected and stored in accordance with the Wikimedia Foundation's privacy policy and data retention guidelines. No IP addresses were obtained.
Results
[edit]A detailed work log with results is at phab:T128536. Below is an overview over the most important results:
_on_mobile_Wikipedia_with_sections_initially_collapsed_vs_expanded,_2016-01-02..08.png)
Readers in the test group (sections expanded) tend to stay longer on the page
[edit]- More detail: phab:T128536#2350536
The pageview session duration (defined as the time between opening the page and the last recorded action by the reader) in the test control group was considerably longer than in the control group, both for the 90th percentile (see chart) and the 75th percentile.
Readers in the test group tend to spend more time reading, and less time navigating
[edit]- More detail: phab:T128536#2383899
The additional time readers spent in the test (uncollapsed) condition could, for example, simply be the result of needing longer to navigate to the desired part of the page. We decided to look at the time a reader spent reading vs. navigating on a page. A timespan between events that are 5 seconds or more apart is assumed to consist of reading, and if subsequent events are 4 seconds or less apart, they are assumed to form part of a continued navigating activity. (I.e. the time the reader spends on a page is split into alternating periods of reading and navigating, with navigating periods ending with a scroll/open event that is not followed by another one within 4 seconds.) The result does not support this concern: The takeaway is that readers in the test condition spend considerably more time reading, and somewhat less time navigating.
Time spent reading a particular page:
| condition | 90th percentile | 75th percentile | median |
| test (expanded) | 190 sec | 53 sec | 6sec |
| control (collapsed) | 146 sec | 38 sec | 5sec |
Time spent navigating a particular page:
| condition | 90th percentile | 75th percentile | median |
| test (expanded) | 4 sec | 1 sec | 0 sec |
| control (collapsed) | 6 sec | 3 sec | 0 sec |
.png)
Readers in the test group tend to scroll more sections into view than readers in the control group open
[edit]- More detail: phab:T128536#2384068
This comparison is somewhat apples-to-oranges (for example because opening a section is normally a deliberate decision based on the content of its headline, whereas - almost per definition - the reader doesn't yet know what a section is about when it is scrolled into view). With all that in mind though, it is still reasonable to interpret this result as a strong indication that readers in the test group are more engaged with the content below the (always expanded) lead section.
Readers in the test group tend to stay shorter on the page than readers using the Android Wikipedia app
[edit]- More detail: phab:T128536#2384336
We also compared the above results about pageview session duration for the test group with existing data for the Android app (which has a similar setup with no sections collapsed). With some caveats about daily/weekly variations and differing ways to measure (the web data does not count the time after the last section open/scroll event when the user is still reading the page before leaving it), it looks like app readers tend to stay on a page longer than readers on mobile web with sections uncollapsed (30-40 seconds at the 75th percentile, 10-20 seconds at the 90th percentile). Interpretations of this result will need to consider that the app offers a floating table of contents for easier navigation, and that app users tend to be more engaged readers to begin with.
.png)
.png)
_on_mobile_Wikipedia_with_sections_initially_collapsed_vs._expanded.png)