IP Editing: Privacy Enhancement and Abuse Mitigation/IP Editing Restriction Study
 | This page is a project write-up by the Anti-Harassment Tools Team to share more details about the IP Editing Restriction Study formerly Login Required Experiment. For related projects, please see page footer. |
Background of the IP Editing Restriction Study
[edit]WMF leadership is currently conducting the IP Editing Restriction Study to monitor the impact of IP Editing restrictions in communities that vote to block IP Editing. So far, we have been studying the following wikis:
- Portuguese Wikipedia (Read the full report)
- Farsi Wikipedia (Read the full report)
Below are summaries:
Portuguese Wikipedia
[edit]- Portuguese Wikipedia voted and turned off IP Editing in October 2020.
- The Anti-Harassment Tools Team have been collecting metrics on the project since then (to June 2021) to understand how this change has impacted the overall state of the wiki.
- A comprehensive report of all the metrics (with graphs) has been published.
- The key areas we focused on include: impact on content, impact on editors, impact on administrative actions and impact on community.
Farsi Wikipedia
[edit]- Farsi Wikipedia voted and turned off IP Editing between October 20, 2021, and April 20, 2022.
- The Wikimedia Foundation's Product Analytics team tracked the impact of this change on Farsi Wikipedia on a weekly basis in order to learn more about the impact of IP editing on a Wikipedia project.
- A comprehensive report of all the metrics (with graphs) has been published.
- The key areas we focused on include: impact on content, impact on editors, impact on administrative actions and impact on the community.
Takeaways from the study
[edit]Portuguese Wikipedia
[edit]We found no significant negative impact in the analysis conducted on Portuguese Wikipedia thus far. However we will continue to monitor for long-term effects for 1-2 years and even more if necessary. Key takeaways are captured below:
- The number of user account registrations as well as active editors on Portuguese Wikipedia have gone up since the IP editing block was introduced. We do not know if it is a long-term trend.
- The user retention metric has not seen a significant increase or decrease. We do not know the long-term impact on it.
- The number of reverts, page protections and blocks have declined considerably, indicating a decrease in the amount of vandalism on the project.
- The total number of edits made to the project show a decline when bot edits and reverts are included. Without bot edits and reverted edits, the number of edits was down in Q2 and Q3 compared to last year, and was up in Q4. Impacts on edits are unclear.
- We also sought community feedback on Portuguese Wikipedia regarding their experience and feedback over the IP editing ban. We found that a majority of the Portugese Wikipedia editors remain in favor of the ban and report seeing less conflict and hostility on the platform.
You may find the comprehensive report for Portuguese Wikipedia's restriction.
Farsi Wikipedia
[edit]On the outset we should note that the Farsi and Portuguese Wikipedia experiments were very different in their structures. Portuguese community chose to block unregistered editors from all namespaces except the Discussion and Help namespaces. On the other hand, Farsi community chose to block unregistered editors from the Main/Article namespace only. The other important factor to note before reading the metrics is that the pandemic has played out differently in different parts of the world and has made a significant impact on metrics across the board.
Highlights
[edit]We can say that the restriction on Farsi effectively reduced vandalism on the wiki. We can say this based on the fact that reverts were down 68% compared to the previous six months and down 70% compared to same time period last year. Blocks were also down by over 50% in both comparisons. This trend was consistent with the Portuguese Wikipedia experiment.
However, the restriction also prevented good-faith edits. The total number of content edits was down -24% compared to the previous six months. This was a much larger decline than what we saw in Portuguese. On Portuguese, content edits declined -15% over the unusually high number in the previous six months, but it was generally in line with the previous couple of years. Farsi's decline in edits was well below the previous years.
Unlike Portuguese Wikipedia, the partial restriction on Persian Wikipedia did not drive new account creation or active logged-in editors.
A survey of Persian Wikipedia editors found that about two-thirds of the editors surveyed had a positive opinion about the experiment, while the rest felt that it was harmful to the project.
Our recommendations
[edit]Based on what we learned, the Wikimedia Foundation's Product department has made the following recommendations:
- Allow Portuguese Wikipedia to continue disallowing IP editing, if the community wants to continue.
- Validate these findings by running a similar experiment on two more Wikimedia projects (languages) where the volunteers are interested in trying this out. The experience of a single wiki should not be used as decisive evidence.
Continuing the study
[edit]Our learnings for Portuguese Wikipedia have been mostly positive. This however should not be used to conclude that we will see similar results on different projects. If you look at the numbers for IP editing we see much varied engagement from unregistered users on different projects. Different projects also behave very differently with unregistered editors.
There is also a separate study – Research:Value of IP editing – published by a group of scholars and researchers who have closely studied Wikipedia which states that there can be lasting negative consequences of blocking edits from unregistered editors.
To understand how other projects would be impacted if IP editing is blocked, we would like to conduct a similar experiment on two more Wikimedia wikis. Here is the proposed plan:
Plan
[edit]- We are inviting communities that are interested in such an experiment on their wiki to express their interest on the talk page. There needs to be some demonstrable historical interest on the project to disallow unregistered editing.
- For each wiki community that is interested, they need to identify a point person who we will be working with to launch and conduct the experiment.
- Community consensus required before the experiment can begin.
- For a time period of eight months, unregistered editing will not be allowed on the selected projects. Over this time, we will collect metrics and do a survey of community members to understand the impact.
- Once this time period concludes, we will turn on IP editing on the project while we prepare the analysis from the data.
- Once the data is published, the communities will be able to decide for themselves whether or not they want to continue to disallow unregistered editing on the project.
Criteria for selection of wikis
[edit]The Anti-Harassment Tools Team will work with two wikis we believe would be best suited to run this experiment based on factors like
- Size of project
- Amount of unregistered editing
- Number of admins
- Amount of vandalism
- How localized or globalized the language is
Timeline
[edit]Timelines have been updated.
| Date | Activity |
|---|---|
| August 24 - October 31 | Communities express interest in running the experiment |
| To be decided after community interest is established. | Anti-Harassment Tools team decides which wikis could proceed with the experiment |
| To be decided after communities decide how long a local RFC will take. | Communities conduct local RfC and agree to run the experiment |
| To be decided when RFC ends | Run the study |
We are looking forward to working with the communities. Anyone can help share this call for participation.
Some Frequently Asked Questions
[edit]Q: Can you give me a background of this "IP Editing Restriction Study"?
[edit]A: Portuguese Wikipedia (ptwiki) decided to turn off IP editing by community decision.
Following the vote, Portuguese Wikipedia initially decided to block IP edits using an abuse filter which was later changed in the favor of using site JavaScript to avoid privacy concerns.
We analyzed the outcome of the change on the health of Portuguese Wikipedia over a period of eight months. The outcome was fairly positive and encouraged us to continue this study on other projects which have expressed a strong desire to block IP editing.
Q: What was the Wikimedia Foundation's role in the study?
[edit]A: WMF leadership and the board wanted to monitor the impact of the change, so the Wikimedia Product Analytics team built a dashboard to monitor the metrics and summarize the data quarterly.
The report from the study was published on meta.
Q: Why did we share the metrics summary?
[edit]A: We believed the community would be interested in seeing the data, and that it may inform continued discussions across the movement about the role of IP editing. Therefore, our team shared the metrics summary on Meta-Wiki for your information without any intention to advocate one way or the other.
Q: How did Wikimedia Product select and define the metrics?
[edit]A: We started with core metrics which this change might impact. Most of them adopted the definitions from this glossary.
For reverts - we used a 48-hour revert window to make the year over year comparisons more valid. If we did not limit reverts to a revert window, edits made last year would have more time & opportunity to be reverted than edits made this year.
Similarly, when we calculated net edits, we only removed edits that were reverted in that 48-hour revert window,
We also further adjusted net edit metrics by excluding edits which reverted other edits in a 48-hour window based on community members' suggestions on Phabricator.
Q: What percentage of users gave up editing instead of creating an account is essential to know. Why is that not captured in the report?
[edit]A: There is no good way to calculate the percentage of users who gave up editing after banning IP editing. Anonymous users can edit from multiple IP addresses and devices. Multiple anonymous users can share one IP address or device.
We cannot identify the number of non-bot editors who edited in anonymous mode before the banning. If we have the data on the unique device of editors, maybe we can alternatively estimate the change of the unique editor devices. However, the foundation does not track unique devices of editors. We have only aggregated unique devices estimation on readers. Please find more info about the unique device tables here.
Q: How do the edits metrics of the ptwiki compare with the other wikis?
[edit]A: We initially compared Portuguese Wikipedia with some other European and South American language Wikipedias, like German Wikipedia, French Wikipedia, Italian Wikipedia, Russian Wikipedia, Spanish Wikipedia.
You can find the non-reverted non-bot edits graphs of the other wikis at the appendix section (page 47-51, page 62-69) in these report slides.
The growth on the other Wikipedias is not present on Portuguese Wikipedia.
These graphs show non-reverted non-bot edits on these European and South American language Wikipedias grew solidly in 2020 while Portuguese Wikipedia did not.
However, after looking at the historical data (net non-reverted non-bot edits), we found Portuguese Wikipedia’s trend does not correlate very well with these Wikipedias. Portuguese Wikipedia is not similar to any other medium-sized wikis. Zooming out to a longer time period by looking at the historical trend (please see page 23 in the slides), non-bot edits (including reverts) on Portuguese Wikipedia have been slightly declining since as early as 2018 while the other European and South American language Wikipedias did not decline.
By comparing Portuguese Wikipedia to the other wikis, we assume that Portuguese Wikipedia would grow as the other wikis do. Historical trends do not support this assumption, that is why we did not discuss it in the report and only put the comparison graphs in the appendix.
We also explored edits (including reverts) by countries to find comparison benchmarks. 95% of edits on Portuguese Wikipedia are from Brazil and Portugal. It has a good correlation with the edits on English Wikipedia from Brazil and Portugal. However, the data is only available for a short period (90 days). We don't have historical data to compare this to.
We also compared bot edits trend on ptwiki to the other Wikipedias, because we see bot edits decreased in 2013 in the historical trend of ptwiki (please see page 23 in the slides). We found it was a common phenomenon across wikis, not the one only ptwiki has.