Research:Editor milestones
Background and rationale
[edit]A number of studies have suggested that barnstars and other onwiki rewards motivate Wikipedians to contribute. However, due to the size of the editing community on English Wikipedia, many editors' good work goes unnoticed and therefore unrewarded. To learn more about the population of editors who are on the verge of becoming "established" Wikipedians, and to test the effects of personalized recognition on editing behavior, in April 2012 we began tracking and thanking all users who reach their 1,000th edit to articles on English Wikipedia.
Methodology
[edit]Using a slave database of the English Wikipedia, we began logging the number of times that users on English Wikipedia made their 1,000th edit to the article namespace (including deleted revisions). We then built an analytics dashboard that automatically updates with the username and timestamp of each 1,000 edit event.
We chose 1,000 edits somewhat arbitrarily, with the rationale that...
- it is a significant contribution to the encyclopedia, and therefore inherently praiseworthy
- it is highly unlikely to be a milestone reached by vandals, spammers, or other overtly disruptive editors
- it suggests a possible "power-user" or very highly active Wikipedian in the making
Starting on April 5, 2012, we began thanking each person who reached this milestone with a personalized barnstar, using WikiLove.
First iteration
[edit]
Currently, about 5-6 Wikipedians cross the 1,000-edit threshold each day. In our first month of performing this experiment, we thanked about 160 editors and analyzed their contribution history to find when they began editing and what subjects they edited most frequently. We then analyzed their editing activity in the 3-day period after receiving a barnstar for reaching their 1,000-edit milestone.
Age of editors (yearly cohorts)
[edit]In addition to what they were currently editing, we also looked at what year these Wikipedians made their first contributions. What we found was that these editors roughly fell into two camps: slow and steady editors who began contributing five or six years ago and accumulated their 1,000 edits over a long period of time, and new users who were among the most active Wikipedians in their first few months of editing and quickly ratcheted up their 1,000 edits.
As demonstrated by the chart to the right, the new Wikipedians outnumber the old – 59% made their first edit in 2009 or later, out of 158 editors to date.
Editing patterns - Pre-Barnstar
[edit]
In order to write a personal thank-you message to each editor, we looked at their user pages and contribution histories to determine what subjects they had been most actively working on. What we found was that many of these Wikipedians were intensely focused on one topic or editing activity. The chart on the right illustrates the most common topics and the percentage of users whose edits mostly fall into each topic. These topics were:
- Sports (e.g., football teams, car races, rugby players)
- Music (.e.g., songs, albums, composers)
- Milhist (e.g., battles, military technology)
- Aviation (e.g., airports, air lines)
- Video games
- TV/film/radio
- Bios (e.g., biographies of people, living or dead, not necessarily connected by any particular topic)
- Local culture (e.g., notable places, events, and cultural phenomena in the city/country where the editor is from)
- World culture (e.g., notable places, events, and cultural phenomena in different parts of the world, not where the editor is from)
- Math/science (e.g., theorems, astronomy)
- Technical gnoming (e.g., copyediting, disambiguating/repairing links, adding navboxes)
- Lit/comics (e.g., novels, manga, comic books)
- Vandal-fighting/patrolling
- Other/varied (e.g., a narrow topic not covered above, such as solar power, or diverse editing activity in many subject areas)
Somewhat surprisingly, by far the most common topic of editing for this sample of users was local culture. Whether they self-identified as being from Turkey, Bosnia-Herzegovina, China, the U.S., or the Philippines, these editors tended to create and/or improve articles about their home town, state, or country.
The other common area of interest for this sample of users was popular culture – in the form of television, film, sports, and music. It is also interesting to note that some of the topics covered by the more active and successful WikiProjects, such as military history, aviation, and biology, were either very rare or not found at all in this sample.
Editing Patterns - Post-Barnstar
[edit]- Mean # of edits in 3-day period after 1k event per user: 49.85
- Mean # of blocks in 3-day period after 1k event per user: 0.01 (1%)
- Mean # of warnings in 3-day period after 1k event per user: 0.24 (24%; however, a few users were warned several times)
From the low rate of blocks and warnings and the relatively large number of edits in the 48 hours after these editors are given a barnstar, we can assume this class of editors continues to be of high quality. However, in order to determine whether a barnstar was the cause of an increase or decrease in activity, a control group must be developed.
Results
[edit]Here's a supplemental analysis comparing the prevalence of edits over three-day periods, both before and immediately after the barnstar was delivered:
http://meta.wikimedia.org/wiki/User:Renklauf/Barnstarred_Editors_Analyses
Second iteration
[edit]Because we are finding patterns in editor behavior that may suggest that thanking editors (via barnstar) for reaching milestones may have a significant impact on their editing, we will iterate on this research by randomly assigning users to a control (no barnstar) and test (barnstar) group. We will then analyze their contributions to determine the following:
Short-term editing effect
[edit]- Control group
- Average number of contributions per user in the 0-3 day period before 1k edit timestamp
- Average number of contributions per user in the 0-3 day period after 1k edit timestamp
- Averages for new editors (2009-2012) vs old editors (2002-2008)
- Test group
- Average number of contributions per user in the 0-3 day period before barnstar timestamp
- Average number of contributions per user in the 0-3 day period after barnstar timestamp
- Average number of contributions per user in the 0-3 day period before 1k edit timestamp
- Average number of contributions per user in the 0-3 day period after 1k edit timestamp (previous month's cohort was 49.85)
- Averages for new editors (2009-2012) vs old editors (2002-2008)
Long-term editing effect
[edit]- Control group
- Average number of contributions per user in the 0-30 day period before 1k edit timestamp
- Average number of contributions per user in the 0-30 day period after 1k edit timestamp
- Test group
- Average number of contributions per user in the 0-30 day period before barnstar timestamp
- Average number of contributions per user in the 0-30 day period after barnstar timestamp
- Average number of contributions per user in the 0-30 day period before 1k edit timestamp
- Average number of contributions per user in the 0-30 day period after 1k edit timestamp
Results
[edit]Short-term editing effect
[edit]This experiment concerns the measurement of revision activity in all namespaces in the context of the following events:
- 1K Event - the time at which an editor reaches their 1000th edit
- Barnstar Event - the time at which the 1K editor received a Barnstar from WMF recognizing their accomplishment of 1K edits
Below is the list of control users and the timestamp of their 1K event:
List of 1K Users - Control (1K timestamp)
|
|---|
ViriiK 2012-05-15 02:58:25 Treydavis3 2012-05-15 06:00:16 Listmeister 2012-05-15 21:12:27 Mike Schwartz 2012-05-16 06:58:17 Horhey420 2012-05-17 19:15:05 Adjwilley 2012-05-18 04:58:54 Loginnigol 2012-05-18 11:39:58 Olaff 2012-05-18 16:47:09 Shanghainese.ua 2012-05-18 20:13:15 Smokeyfire 2012-05-19 00:53:09 Wikimace08 2012-05-19 21:35:27 Acsian88 2012-05-20 02:31:29 Jackl 2012-05-20 08:12:00 DonBarton 2012-05-20 19:57:03 Abh87 2012-05-20 21:03:52 AzseicsoK 2012-05-21 01:20:31 Bazuz 2012-05-21 12:08:57 Inscription 2012-05-21 16:41:46 Ttenchantr 2012-05-21 18:32:56 Dzlinker 2012-05-21 22:48:32 Kwesiidun91 2012-05-22 02:30:50 Dougsim 2012-05-22 12:39:56 Gwen-chan 2012-05-22 15:37:50 Room429 2012-05-22 18:01:19 Ekin 2012-05-23 12:09:20 JoelWhy 2012-05-23 22:00:59 M0rphzone 2012-05-24 01:46:09 KarlcmanRockets 2012-05-24 11:48:10 Ich 2012-05-24 16:52:41 Tyrannus Mundi 2012-05-25 18:40:27 Callanecc 2012-05-26 02:21:46 BrightStarSky 2012-05-26 05:22:40 Allkayloh 2012-05-26 13:46:11 Paulika1995 2012-05-27 11:43:46 Trafalk09 2012-05-27 20:52:31 TRLIJC19 2012-05-28 04:17:44 PaulWicks 2012-05-28 12:34:14 Beefyt 2012-05-29 06:02:32 Draganmkd 2012-05-29 11:50:16 MCTales 2012-05-29 23:43:51 Tyrogthekreeper 2012-05-30 09:20:44 JmKissme 2012-05-30 16:41:03 Soulboost 2012-05-31 00:15:19 Gbawden 2012-05-31 13:06:14 Meagles111 2012-06-01 03:33:38 EllsworthSK 2012-06-01 20:39:15 SecretStoryStyle 2012-06-01 22:15:47 MaxPayne888 2012-06-03 03:04:23 Healy10 2012-06-03 14:20:28 Mhygelle 2012-06-03 18:12:41 Sorie 2012-06-03 22:32:59 Preacher lad 2012-06-04 08:25:58 Dark Silver Crow 2012-06-04 15:49:10 Zhinz 2012-06-05 02:35:28 Karin127 2012-06-05 10:17:46 Hornymanatee 2012-06-06 10:31:02 FoCuSandLeArN 2012-06-06 15:13:28 Anarchangel 2012-06-06 23:33:58 Anir1uph 2012-06-07 17:26:18 M samadi 2012-06-08 11:00:34 Spparky 2012-06-08 13:42:23 Writerchic99 2012-06-08 21:35:45 Rudyindarto 2012-06-09 07:17:40 Bar Code Symmetry 2012-06-09 15:49:14 Tchaliburton 2012-06-09 22:30:18 Loupiotte 2012-06-10 00:47:59 Solarra 2012-06-10 16:03:34 Samil20 2012-06-11 05:41:41 Lovysinghal 2012-06-12 08:59:57 Treki 2012-06-12 13:25:16 |
Below is the list of test group editors and the timestamp of their 1K event:
List of 1K Users - Test (1K timestamp)
|
|---|
3210 2012-05-15 02:24:56 Werothegreat 2012-05-15 15:33:56 Tholme 2012-05-15 20:31:59 Yutsi 2012-05-15 22:06:04 Andy4789 2012-05-17 21:00:10 Redhanker 2012-05-18 07:16:18 Nikpapag 2012-05-18 12:29:32 Yachty4000 2012-05-18 20:18:23 Anderch 2012-05-19 09:57:31 BuzyBody 2012-05-20 00:17:06 Thepidding 2012-05-20 03:42:15 Danrok 2012-05-20 13:54:27 Rui78901 2012-05-20 21:01:46 Peetlesnumber1 2012-05-20 21:35:15 Riley Huntley 2012-05-21 03:35:55 Alborzagros 2012-05-21 12:27:12 Yahnatan 2012-05-22 00:38:41 Jonadin93 2012-05-22 02:58:11 Jprg1966 2012-05-22 14:51:18 Editfromwithout 2012-05-22 22:41:12 Coal town guy 2012-05-23 14:05:36 Sir Wrestler 2012-05-23 23:07:22 Treyvo 2012-05-24 04:20:56 JZCL 2012-05-24 16:43:38 Idiotchalk 2012-05-24 17:18:49 JeanneMish 2012-05-25 13:23:52 Entity of the Void 2012-05-25 23:40:58 EricEnfermero 2012-05-26 02:55:56 Xtian06 2012-05-26 12:23:46 Gylatshalit 2012-05-26 19:19:51 EagerToddler39 2012-05-27 03:53:25 Devite 2012-05-27 15:56:02 Ericleb01 2012-05-28 01:14:50 Jpaulm 2012-05-28 22:45:17 Jonathan A Jones 2012-05-29 07:15:54 Phileasson 2012-05-29 11:59:18 Dorje108 2012-05-30 02:56:49 Welshsocialist 2012-05-30 22:21:45 Tthaas 2012-05-31 00:16:20 TheSpecialUser 2012-05-31 16:35:17 Ashok2102 2012-05-31 16:49:22 David1217 2012-06-01 03:14:30 DeXXus 2012-06-01 07:56:18 Jayven09maddie 2012-06-01 18:28:31 Russ Jericho 2012-06-01 20:43:16 Witger 2012-06-02 09:29:45 Rytyho usa 2012-06-02 22:42:41 Narcis90 2012-06-03 13:15:09 SolomanMcKenzie 2012-06-03 16:29:07 Cloptonson 2012-06-03 20:24:21 Fylbecatulous 2012-06-04 02:08:04 TuneyLoon 2012-06-04 20:17:31 Jfd34 2012-06-05 03:31:44 RJR3333 2012-06-05 21:31:08 Xxhopingtearsxx 2012-06-06 22:14:34 Boseritwik 2012-06-07 05:35:46 Total-MAdMaN 2012-06-07 22:41:52 FallenAngelII 2012-06-08 00:12:20 Sidsahu 2012-06-08 12:50:07 Carcamagnu 2012-06-09 06:41:29 Run34 2012-06-09 20:07:35 CodeCat 2012-06-09 22:57:26 TheTimesAreAChanging 2012-06-10 22:29:30 VarietyPerson 2012-06-11 14:24:18 Bearpatch 2012-06-12 00:47:29 |
Below is the list of test group editors and the timestamp of their Barnstar event:
List of 1K Users - Test (1K timestamp)
|
|---|
Werothegreat 2012-05-16 16:33:00 3210 2012-05-16 17:01:00 Yutsi 2012-05-16 17:08:00 Tholme 2012-05-16 18:00:00 Andy4789 2012-05-18 18:07:00 Redhanker 2012-05-18 18:10:00 Nikpapag 2012-05-18 18:17:00 Alborzagros 2012-05-21 19:15:00 Riley Huntley 2012-05-21 05:40:00 Peetlesnumber1 2012-05-21 19:28:00 Rui78901 2012-05-21 19:59:00 Danrok 2012-05-21 20:02:00 Thepidding 2012-05-21 20:05:00 BuzyBody 2012-05-21 20:10:00 Anderch 2012-05-21 20:15:00 Yachty4000 2012-05-21 20:18:00 Yahnatan 2012-05-22 15:58:00 Jonadin93 2012-05-22 16:02:00 Jprg1966 2012-05-22 06:08:00 Idiotchalk 2012-05-24 22:30:00 JZCL 2012-05-24 22:32:00 Treyvo 2012-05-24 22:35:00 Sir Wrestler 2012-05-24 22:37:00 Coal town guy 2012-05-24 22:41:00 Editfromwithout 2012-05-24 22:44:00 Tow 2012-05-30 18:46:00 Cyan Gardevoir 2012-05-30 18:48:00 JeanneMish 2012-05-30 18:53:00 Entity of the Void 2012-05-30 19:02:00 EricEnfermero 2012-05-30 19:06:00 Xtian06 2012-05-30 19:19:00 Gylatshalit 2012-05-30 19:22:00 EagerToddler39 2012-05-30 19:58:00 Devite 2012-05-30 20:18:00 Ericleb01 2012-05-30 20:23:00 Jpaulm 2012-05-30 20:49:00 Jonathan A Jones 2012-05-30 23:19:00 Phileasson 2012-05-30 23:41:00 Dorje108 2012-05-30 23:44:00 Welshsocialist 2012-05-31 21:54:00 Tthaas 2012-05-31 22:11:00 TheSpecialUser 2012-05-31 22:15:00 Ashok2102 2012-06-01 22:20:00 David1217 2012-06-01 22:26:00 DeXXus 2012-06-01 22:30:00 Jayven09maddie 2012-06-01 22:34:00 Russ Jericho 2012-06-01 22:43:00 Witger 2012-06-04 19:12:00 Rytyho usa 2012-06-04 19:16:00 Narcis90 2012-06-04 19:19:00 SolomanMcKenzie 2012-06-12 17:45:00 Cloptonson 2012-06-04 19:29:00 Fylbecatulous 2012-06-04 19:48:00 TuneyLoon 2012-06-05 18:50:00 Jfd34 2012-06-05 18:59:00 RJR3333 2012-06-07 18:42:00 Xxhopingtearsxx 2012-06-07 18:48:00 Boseritwik 2012-06-07 18:51:00 Total-MAdMaN 2012-06-08 18:47:00 FallenAngelII 2012-06-08 18:52:00 Sidsahu 2012-06-08 18:55:00 Carcamagnu 2012-06-11 20:03:00 Mogism 2012-06-11 20:06:00 Run34 2012-06-11 20:08:00 CodeCat 2012-06-11 20:13:00 TheTimesAreAChanging 2012-06-11 22:25:00 VarietyPerson 2012-06-11 22:28:00 Bearpatch 2012-06-12 23:50:00 |
Editor Response: behaviour after event
[edit]These are preliminary results showing a comparison of the revision activity in the 0-3 day period after (1) the test 1K event and control 1K event and (2) the test barnstar event and control 1K event. The results are inconclusive.
T-Test results - Control 1K vs Test 1K
|
|---|
data: metrics_1K_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 t = 0.0575, df = 132.695, p-value = 0.9542 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -44.85078 47.53869 sample estimates: mean of x mean of y 69.81538 68.47143 |
T-Test results - Control 1K vs Test Barnstar
|
|---|
data: metrics_bStar_test_all$revisions_after_0_3 and metrics_1K_control_all$revisions_after_0_3 t = -0.4677, df = 121.68, p-value = 0.6409 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -51.10443 31.57334 sample estimates: mean of x mean of y 58.70588 68.47143 |
Editor Response: behaviour before and after event
[edit]In addition to measuring the revision activity of the editors in this experiment in 0-3 day period after the events the revision activity in the 0-3 day period before the event was also measured:
- Edit Diff = edits_0-3_days_after - edits_0-3_days_before
Below are the results of the T-tests over this metric, which yielded no significant results between the groups.
T-Test results - Edit Diff - Control 1K vs Test 1K
|
|---|
data: metrics_1K_test_all$edits_diff and metrics_1K_control_all$edits_diff t = 1.2269, df = 101.934, p-value = 0.2227 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -17.85703 75.77571 sample estimates: mean of x mean of y 7.430769 -21.528571 |
T-Test results - Edit Diff - Control 1K vs Test Barnstar
|
|---|
data: metrics_bstar_test_all$edits_diff and metrics_1K_control_all$edits_diff t = 0.2081, df = 96.076, p-value = 0.8356 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -41.04367 50.65964 sample estimates: mean of x mean of y -16.72059 -21.52857 |
Long-term editing effect
[edit]The results when observing the 30 day period after the barnstarring yielded the same results as the short-term effect. That is, no significant difference in editing behavior was observed among the two groups.
T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar
|
|---|
Welch Two Sample t-test data: metrics_1K_test_all$edits_30 and metrics_1K_control_all$edits_30 t = -0.1107, df = 124.506, p-value = 0.912 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -260.2132 232.6440 sample estimates: mean of x mean of y 421.4154 435.2000 |
T-Test results - Edit Diff 30 - Control 1K vs Test Barnstar
|
|---|
Welch Two Sample t-test data: metrics_bstar_test_all$edits_30 and metrics_1K_control_all$edits_30 t = 0.5423, df = 133.347, p-value = 0.5885 alternative hypothesis: true difference in means is not equal to 0 95 percent confidence interval: -217.9296 382.5590 sample estimates: mean of x mean of y 517.5147 435.2000 |
Raw data
[edit]Full data set is available here.
Conclusions
[edit]Being awarded a barnstar for making 1,000 edit to the article namespace does not appear to have a significant effect on a English Wikipedians' short-term or long-term editing activity. More work may be necessary to determine whether a different barnstar metric (e.g., more or less than 1,000 edits, a certain number of edits in a successive editing session, etc.) or a different awarding treatment (e.g., community-awarded barnstars, rather than staff-awarded) produces an effect.
Discussion and Follow-up
[edit]- Impact on other experiments
These results may inform post-registration activities we experiment with in our account creation UX work. If editors likely to be very productive in the encyclopedia edit either base on their geographic location, cultural heritage, and hobbies, then suggesting this may be fruitful. It may also tie into the work proposed on structured profiles that can be used to suggest articles to edit and WikiProjects to join.
- Other research questions
- Previous research efforts looked at dates 87-90 pre-milestone. We should look at the average contributions per user in the 0-3 day period 90 days before the barnstar, to account for any possible seasonal effects.
- Review whether the findings from Aaron Shaw's and Mako's study on barnstars holds true. (They found that a burst of activity could usually be observed in editors' contribution histories immediately before receiving a barnstar, followed by a decline in activity.)
- Does receiving personalized recognition instill a stronger community sense in new Wikipedians? Analyze userpage creation and edits in the non-article space for test and control groups.
- How does the status of the giver affect the outcome of the barnstar? Perform another month-long A/B test, asking members of the community to award barnstars to the test group and continue staff barnstarring as a control.
- How has the editing pattern of 1k editors changed in the past year? Pull a list of editors who achieved the 1k milestone 1 year ago and compare to control group. Analyze whether the cohort graph looks the same but shifted back one year (e.g., did most people who reached 1k edits last year begin editing in 2008?)
- Other A/B test ideas
- Test sending the barnstar from a volunteer account rather than a WMF one. (Shaw and Mako's study suggested that the sender of a barnstar matters as much as the barnstar itself, so a thank-you message from another editor may have a different effect than one from a staff member.)
- Test removing mention of 1,000 edits and only including a personalized thank-you message. (Does priming with edit count create editcountitis, does it lead to more/less contributions than no priming?)
- Test simple talkpage message vs WikiLove. (Any kind of talk page activity might lead to a short-term bump in contributions, so we should see whether it matters what the message is.)
See also
[edit]- Research:Post-edit feedback#Experiment #2 - Historical Feedback - providing specific feedback when users reach 1, 10, 50, 100-edit milestone
- /Lifecycle - more analysis on the lifecycle and editing velocity of the users in the sample