Grants:Project/Daimona Eaytoy/AbuseFilter overhaul/Timeline

This project is funded by a Project Grant

Timeline for Daimona Eaytoy

Timeline	Date
Architecture overhaul	15 November 2020
Bugs fixed, shared variables implemented	31 December 2020
Any stretch goal	15 January 2021

Monthly updates

Please prepare a brief project update each month, in a format of your choice, to share progress and learnings with the community along the way. Submit the link below as you complete each update.

September (half month)

The project began as expected on Sept. 15. We set up everything needed for us to work on the project (metrics, communication channel, phabricator workboard). We went through a list of AbuseFilter bugs and decided which ones to work on. This list can be seen on the workboard.

We decided to divide the project into two big units: architecture review + tests + namespaces, and bugfix + new features, and we immediately began with the former. Some tests were written for hot and uncovered pieces of code; other irrelevant pieces of code were excluded from reports. The coverage itself didn't increase much (it was around 39% by the end of September), but that's going to change later on. One point worth noting is that part of the work is currently blocked by T246539: we need a sysadmin to finish running the script before we can touch some delicate parts of the codebase.

We agreed on a few delicate topics re architecture review, and began writing the "new" code. The boilerplate code for services was put in place, and we're now building on top of it, adding new services.

Details about the goals:

As per above, coverage is growing slowly for now (it started from ~35%)
We're already adding namespaces, but most of these are temporary for now (and until more code will be migrated to the new architecture)
Code coupling is also hard to reduce, especially because the original graphs was not accurate (see r631748, several dependencies were hidden), but the new services will help a lot with this.
We're probably going to fix more than 10 important bugs, but it's still to early to say for sure.

October

In this month, we continued working according to our plans. More specifically:

Test coverage actually went down (from 40 to 35%), but this is not accurate: first of all, some previously-covered lines were not really covered, so we removed them from the count. Second, we added Selenium tests, that are not counted when generating coverage. Every new piece of code is actually covered by tests, so we can say that the coverage has actually increased. It's worth noting that something similar is going to happen if we remove the old parser.
Progress was made with the updateVarDumps, thanks to Martin Urbanec who ran the script on most of the remaining wikis. Only ~15 wikis are missing now, after which we'll be able to refactor some more delicate areas.
The various AbuseFilter classes are now more tangled than ever but this is expected, and it's going to fix itself eventually. We need a bunch of other services (some are already under review), and some of these tangles are created by backwards compatibility code, which is obviously not really used by AbuseFilter itself (it only exists for other extensions, see e.g. phab:T266380.

Overall everything is fine, we might need a bit of extra time to ensure no legacy code is left in place, but things are going pretty smoothly.

November

The update for November is quite short, because most things were already reported in the midpoint report at the same time. A quick overview of the status quo:

Many things were refactored. The next areas to cover are AbuseFilterRunner, the View classes (both of these are under review), and then the VariableHolder-AFComputedVariable tangle once T246539 is done. Then some remainings from the AbuseFilter main class.
After the above is done, we're going to put the finishing touches to tests and namespaces
Several bugs were resolved, although we'll put more focus on this in the second part of the project
As already mentioned multiple times, T246539 is now the biggest source of uncertainty. We might as well leave some refactorings for later, and begin the second part of the project sooner, then waiting until we can refactor the other areas.

December

December has been a great month, because many things were moved forward; in particular, the maintenance script was executed on all WMF wikis (T246539), which means that it's no longer a blocker, and we can refactor that area as well. More precisely:

More code was refactored. We are quite close to the end goal here, with just a bunch of classes missing
Some namespaces were adjusted, also in test classes
Code coverage has increased, being now at 42.31%
More bugs were fixed, including some new features like echo notifications for throttled filters and real-time validation of rules

January

During January, most goals related to code health were reached. In particular:

Code coverage has reached 51.72%, which is higher than the goal of 50%. Tests in general were improved, and although some code is still not covered, the situation is now much better than when we started!
The AbuseFilter "god class" was almost emptied. It now consists of two deprecated methods (used in other extensions), a constant, and an utility method. While not really empty, I think this can be considered on par with the goal of emptying the class: the deprecated methods have to stay; the constant and the utility method might be moved to a separate Utils class, but it wouldn't really make any difference.
Everything is now namespaced. Some of these namespaces might be further improved (that is, some classes might be moved in a dedicated namespace) in the future, but this would require further improving the code first, which is outside of the scope of the grant. So this is also on par with the grant goals.
No existing bugs were actually fixed this month, just several bugs introduced with new code. We'll be focusing on bugs in February.
We've investigated the implementation details for shared variables, and determined that it won't be really possible to deliver a complete product within the project constraints. More about this to come in a few days.

February

As the last month of the project, during February we mostly tweaked a few things around, and improved some portions of code that we marked for improvement earlier during the project. As such, most changes consisted of test or documentation improvements, replacing deprecated methods and the like. The most notable exception to this is some work by Matěj on improving the emergency shutdown system (which is now more precise, and it also has cleaner code) and also improving the main FilterRunner class, which now uses structured value objects to store information, thus making it easier to add new functionality. All in all, the project goals were met as will be outlined in the final report shortly.

Extension Request

Is your final report due but you need more time?

Extension request

New end date

February 28, 2021

Rationale

@Mjohnson (WMF): As somewhat expected, some parts of the overhaul ended up taking more time because of software unpredictability. Things went quite smoothly, and there've been a lot of changes, but the current end date is too close for us to finish everything. I think we might not need the whole period until Feb 28, but that date should allow us to finish the work without rushing. As per the December report, we're almost done with the architecture overhaul part, and some work in the bugfix part was anticipated, so an additional month and a half should suffice for completing the work, and perhaps even some stretch goals. Thank you, --Daimona Eaytoy (talk) 14:41, 3 January 2021 (UTC)

Request approved

Dear Daimona Eaytoy,

This request is approved. Your new end date for your project is February 28, 2021.

Warm regards,

--Marti (WMF) (talk) 22:14, 15 January 2021 (UTC)

Noting here that the new final report due date is 30 March 2021. -- JTud (WMF), Grants Administrator (talk) 22:16, 15 January 2021 (UTC)