Wikimedia Fellowships/Proposals/Evaluating welcome messages

Proposal for a Wikimedia Fellowship, where the fellow would develop a tool to evaluate the success of welcome messages on Wikimedia projects

Background

Wikimedia is a project that relies on volunteer labour, but only a tiny proportion of the volunteers who start editing here continue doing so for any great length of time. Recently the total number of volunteers has peaked and started to fall on many wikimedia projects, this fellowship is intended to halt and reverse this by applying tried and tested marketing techniques to the way in which we treat new editors, with the aim of increasing the proportion of newbies who stay around and become active Wikimedians.

Assuming EN Wikipedia is fairly typical if perhaps larger than many, we have used dozens of different welcome messages to welcome hundreds of thousands of newbies. Different welcome messages and simply the receipt of one may make a big difference as to whether or not a newbie stays, edits constructively or goes away. Discussions as to which welcome message to use tend to be aesthetic rather than scientific.

Treating this as a marketing exercise and evaluating different messages in terms of their longterm effectiveness at getting newbies to become regular editors will either identify messages that work better than others, or prove that the whole welcoming thing is a waste of effort and we are better off not doing it. Either way we have much to gain by measuring the effectiveness of different welcome messages.

Product, goals, and metrics

The fellow would be expected to write or commission code for a tool on the toolserver to enable any wikimedia project to run a comparison process using any of the welcome templates they have used in the past, against the general trend of editor retention on their project.

As many welcoming messages have been used by many wikimedians over time, and they have been used under various unrecorded selection criteria, the measurement of their relative effectiveness is necessarily complex.

At a minimum the testing will need to take account of the following variables:

Different templated welcome messages are likely to have been used over time, in some cases particular welcome templates may themselves have changed. The software needs to be able to identify minor variations based on the edit history of the template.
Different Wikimedia projects or wikimedians will have different definitions of a successful welcome message "total edits in the first 12 months after welcoming per 1000 editors welcomed" could be one measure. But on that measure a welcome message that put off everyone except a huggler who notched up 100,000 edits in their first year then left, would be considered more successful than a welcome message that was given to a bunch of editors who were still around two years later doing 100 edits a month adding referenced encyclopaedic material. "Percentage of editors doing over 100 and over 1,000 edits in the 12 months after receiving the welcome" might be a default criteria to use. But Projects will undoubtedly want the option to tweak this, and ideally to do so by type of newbie welcomed.
Each wikimedia project has a different profile of editor involvement as some projects appear to have become less welcoming over time. This will need analysing by project and a baseline created using historic data on unwelcomed newbies.
Different welcome templates and versions of templates will need identifying from talkpage histories and comparing to the newbie retention baseline for that project
As newbie edits and especially their articles often get reverted or deleted, and welcomes may be blanked or archived, the tool will need to use more than the current visible and undeleted edits, it will also need access to deleted edits.
The speed with which welcomes are applied. The theory is that most genuine welcomes as opposed to warning ones are given by established editors to newbies who they notice making a goodfaith edit. But there may be some editors welcoming everyone who creates an account, before being able to see if they are a vandal
Retention varies dramatically according to what people had already done when they were welcomed. Analysis needs to take account of this, most importantly will probably be the amount of edits the newbie had done before being welcomed, but in some cases the proportion of editors ultimately blocked will be an important measure. For example on EN Wiki as well as welcomes for good faith newbies, there are many others including:

en:wiki/Template:Firstarticle for someone whose first article is being speedy deleted. On EN wiki, article creators whose first article has been deleted are much less likely to continue editing than other newbies.
en:wiki/Template:Welcomespam for spammers.
en:wiki/Template:Welcome-warning - for editors whose first edit is vandalism (success for a welcome template targeted at vandals needs to be compared with other welcome templates issued to vandals, not with the general trend).

The Fellow would need to be able to communicate with diverse projects about this tool, explaining how to use it and how to interpret the results. Crucially they need to explain that this is about finding out which templates work better than others, enabling projects to test that. and thereby enable wikimedians and bots who welcome newbies to make better or more confident decisions as to which templates to use.

Theories worth testing

There is a theory that what makes a difference is whether the welcome is from someone who the newbie knows they have interacted with. So editors who categorise articles or fix typos and welcome newbies they come across may be more effective welcomers than editors who patrol recent changes, warn and revert the vandals and welcome the goodfaith editors.
There are contrasting theories about the use of friendly images such as plates of cookies. One theory is that they make the welcome more empathetic and positive, another is that they make it look more computerised and impersonal, a third is that they add bytes and annoy editors with slow connections or in parts of the world where access to Wikimedia servers is poor.

Bonus features

It would also be useful if the same software could be used to test the effectiveness of various warning templates at deterring or rehabilitating vandals, spammers and so forth.
Direct marketing is usually much more effective if the message is targeted according to the recipient, such as by use of longer copy for a female audience, and by the use of relevant info. A welcome message that took account of the Newbie's gender in copy length and in some cultures phrasing, and also mentioned any wikiprojects that were relevant to the articles they had edited, would require an unusually sophisticated template, but in theory should outperform the best current welcome messages. Developing this would be a worthwhile bonus feature.

Logistics

This would be a Community Fellowship, as it would benefit many if not all WMF projects - even the ones that don't currently have multiple templates to test would benefit from knowing how templates with images of plates of cookies/sweets/local cultural equivalent generally compare to lectures about referencing or lists of pillars.

The fellow would need to communicate with the wikimedia community and write open source software that was available for others to enhance and maintain.

This fellow could be located anywhere that has Internet access.

Fellow(s) responsible for this proposal

If the proposal has community support, then it would be nice if an appropriately qualified person would bid for this fellowship. IT skills and an understanding of wikimedia are the main qualifications.

Applicants

This section is for Wikimedians who would like to undertake this project: (Please state whether you need a grant or could write this as a volunteer)

Endorsements

This section is for endorsements by Wikimedia volunteers. Please note that this is not a debate, vote, or poll, but is rather a space for volunteers to describe in detail why they think a fellowship project is or is not of value. If you have concerns or questions rather than an endorsement to make, please do so here. Endorsements by volunteers willing to work in collaboration with a fellowship recipient on a project are highly encouraged.

As proposal initial author, if someone is prepared to do this I would be happy to collaborate with them. WereSpielChequers 20:08, 11 March 2011 (UTC)[reply]
I don't know how to do coding but I support the proposal and definitely think this is worthwhile. Would be very interested in seeing the results. -- OlEnglish ^(Talk) 02:59, 21 March 2011 (UTC)[reply]
This would be a very useful endeavor, and I would be interested in collaborating on this in whatever was is useful. --NickPenguin 16:21, 21 March 2011 (UTC)[reply]
This can only be a Good Thing™. ←fetch comms 02:05, 11 May 2011 (UTC)[reply]
I think this is an excellent idea. I think we have lots of newbies going away without editing because we have some hills at our doorstep. We need to remove these hills and make their entry easier through the welcome template. Am unfamiliar with coding, but would like to put up some suggestions for consideration.MangoWong 13:41, 24 September 2011 (UTC)[reply]