Research talk:Timestamp position modification

Testing options

Current version has our methodology listed as comparing the edit and view history clickthrough data on the same sample for the same length as the experiment.

Normally the cleaner option would be an A/B split test, where we take all the traffic to the articles in our random sample and present half with the default and half with the test. However, my understanding is that no one has ever done this before, except perhaps through CentralNotice.

Unless I am overestimating the difficulty of building a new A/B testing framework from scratch, I would say we should err on the side of less complex development, since we selected this as a first experiment because it would be easy to have ready to deploy.

In any case, the first revision we wrote based on our initial conversations was this:

"We will compare clickthoughs to the history and edits to the same sample of articles for a time period equal to the length of the test immediately prior to the start of the test." So we look at the same data for a week before we run the test, we run the test for a week, compare.

That way includes one possible confound, which is some current event which spikes traffic and/or edits to a particular article, but seems like it would be the easy route to simply cull such outliers during analysis phase.

Steven Walling (WMF) • talk 05:23, 16 May 2012 (UTC)

Ideas for next iterations

1. For articles that have not been updated in a year or more: change text to deliver a message more like: "This article needs updates!" or another to directly ask for fresh edits.
2. Linking to the diff instead of history, especially for updates shorter than 1 week or 24 hours.
3. Evaluate edit velocity change on classes of articles -- e.g. B Class, stubs, etc.
4. Evaluate edit velocity change on older articles (those in need of update)

Steven Walling (WMF) • talk 18:27, 17 May 2012 (UTC)

Measures

I've read the pages on mw: and the discussion on en.wiki, as well as this page, but I still don't understand what's actually being tested and how the test will be judged. The user stories seem absolutely unrealistic to me and anyway are impossible to measure. The chosen metrics suggest that this would be considered a success if it increases the number of edits, but this seems weird to me: when the page was last updated is obviously an information of very little (or negative) value, what's important is that users learn about the history page and this is what we should measure; it seems highly dubious, and not so relevant, that this will bring more edits to the affected pages, while one should measure if this increases awareness about the history page, the chances that the affected users look at it elsewhere and that they therefore also edit more on other pages because of this knowledge. That said, I understand that this is mainly a way to test a new experiment framework and actual tests, which are mroe difficult, will come later, but it gave me some headache before I got it. --Nemo 06:34, 18 June 2012 (UTC)

User stories are not directly used for any analysis. They're just to give the designers and developers a rough idea of what the user might get out of using a feature. That's all. Anyway, yes we are interested in edits, but you're right to say that it's unrealistic that this would directly produce many new edits. We are aware of that, which is why we are also measuring aggregate clicks on the "View history" tab as well, to compare to clicks on the timestamp. On the one hand, the intangible goal of greater reader awareness about the mutability of pages is difficult if not impossible to measure. On the other hand, measuring how many people use our current functionality for inspecting the history and then comparing that to clickthroughs via the new timestamp is a very effective measure of how many people actually are engaged enough to inspect the page history. Steven Walling (WMF) • talk 17:46, 18 June 2012 (UTC)

I think we should make it clear that edit CTR and conversions are not the focus of the first iteration. Nemo, one of the plans for future iterations on this experiment is to add call to action on the landing page conditional on the referral (You landed on the revision history via the modified timestamp? Here's an invitation to edit the article from the history page) and measure to what extent this produces a higher volume of new editors. We don't expect to see effects on new editor conversion until we have a funnel in place that links the modified timestamp with an invitation to edit. --DarTar (talk) 21:04, 21 June 2012 (UTC)

What's the next step

Probably me just being dense, but what's the actual plan with this idea? Is it going to be rolled out to more people, more permanently? Personally I don't particularly care if it increases the number of hits to the history page, it's just a very useful method for readers to gauge the topicality of the article, give it a sense of immediacy, and make the fact that "people actually wrote this in iterative fashion" more clear. In a very similar vein I've always liked the assessment notification under the article title gadget. Wittylama (talk) 05:51, 22 October 2012 (UTC)

Don't suppose anyone wants to answer this...? From what I can see from the article about this, it can be classified as a success and an easy thing to do. Are there any plans therefore to actually do it? Wittylama (talk) 00:27, 1 February 2013 (UTC)

Sorry about the wait on the answer. There's a duplicate thread on this topic at mw:Talk:Timestamp position modification. Steven Walling (WMF) • talk 00:43, 1 February 2013 (UTC)

You mean the one where you've commented "The short answer is that it definitely seems to be of use to readers and editors, but because our goal is to focus on things that directly aid editing, we're not making it permanent right now."? I'm the last follow-up comment on that thread over there too :-) I'm a really big fan of this idea and would love to see it implemented, if only because it *appears* to be such a simple and obvious idea that *appears* easy to implement. As you know I've no idea about technical requirements but I suspect that it would be trivial to actually enact across all our projects? For something that has had all the testing and planning work already completed (and even announced on the WMF blog) surely it's a waste to then just leave it fallow after having it declared a success by community feedback as well as quantitative measures?! Wittylama (talk) 23:42, 1 February 2013 (UTC)

The truth is almost nothing is trivial to enable across the projects. It'd probably take us at least a week's worth of testing, refactoring, and bug fixes to even just enable in English. Steven Walling (WMF) • talk 23:35, 2 February 2013 (UTC)

I'm not a big fan of the idea. And I don't see how this was declared a success by community feedback as well as quantitative measures? Let's have a closer look:

• Changing the text from "This page was last modified on XXX" to "Last updated XXX days ago" was felt to be a less accurate description ("false advertising").
• The fuzzy time "XXX days ago" instead of the actual date is now applied on mobile Wikipedia (at the bottom, just like the old timestamp position on desktop)
• The prominent placement in the timestamp experiment. I think the Research question "Does a more prominently visible timestamp of the last revision produce a significant increase in clickthrough rates to the history of an article?" led to the rather trivial insight that, indeed, a more prominent link placement gives a higher clickthrough-rate: "The presence of the new timestamp appears to produce twice the amount of clicks on the history by readers or anonymous editors. (...) suggests that readers are in fact aware of the view history tab and what it stands for, and that the more prominent timestamp drove them to inspect the history of an article." Or maybe they were driven rather by the promised "update", who knows?
  • If readers "are in fact aware of the view history tab and what it stands for" then why does it need a further prominent placement? There is the prominently placed link "View history", there is an explicit text at the bottom ("This page was last modified on XXX") and there is the link "page information" on the left under toolbox.
  • Placement conflicts. These have been noted early ([1]). There are many icons for FA, GA, maps, coordinates, sighted version, protected etc. at the top of articles. There are enough placement conflicts as it is, and new ones to come:

• 
  Proposed AFT link at top
• 
  Conflict coordinates and sighted versions
• 
  Conflict long article title and sighted versions
• 
  Prominent timestamp position

• The research justification, that "We expect this experiment to have a minor, indirect impact on editing" wasn't analysed, but in "Future Iterations" it was intended "to iterate on this experiment by including a call to action upon clicking the last modified link". Looking at the placement research by the AFT team (AFT placement analysis and Quality assessment), I doubt that the call to action will lead to helpful edits.

In conclusion: I don't really see the timestamp experiment leading to next steps. --Atlasowa (talk) 18:13, 4 February 2013 (UTC)

The placement conflicts could be worked out, as could issues like the implementation in JavaScript. However, I think Atlasowa is helping me demonstrate that turning an experiment in to a product is almost never simple in MediaWiki... Steven Walling (WMF) • talk 19:53, 4 February 2013 (UTC)

Atlasowa: I've had the last modified date above the article tabs for years for my personal account:

I think it's tremendously helpful to both readers and editors to see when a page was last edited. The technical issues you raise can be remedied, I think. The same probably can't be said for the social issues. That is, Mobile Wikipedia is doing its own separate development and attempting to use this change as a tool for editor retention are both probably not going to be fixed, regardless of whether we add this feature [to the desktop site]. --MZMcBride (talk) 03:38, 28 May 2013 (UTC)