Toolhub/Progress reports/2021-02-12
Report on activities in the Toolhub project for the week ending 2021-02-12.
Security Readiness Review for Toolhub planned
[edit]The Wikimedia Foundation's Security team has accepted Toolhub as a project to be reviewed in the April-June quarter.
Fun times with upstreams leads to fixing tech debt
[edit]pyca/cryptography is a widely used library dependency for Python projects. The cryptography library provides high level abstractions for dealing with a variety of crypto tasks including extensive support for X.509 public key certificates. X.509 is important as core component of TLS/SSL encryption for web traffic.
The 3.4 version of pyca/cryptography released on 2021-02-07 (followed quickly by several point release fixes) introduced a new build time dependency on the Rust programming language. The upstream also provides pre-compiled versions in the Python wheel package format, but in the 3.4 release they also changed the version of that packaging spec that is used for their project.
This combination of upstream changes intersected with a more local change to break CI for us for a while. The local change was actually a bug fix made by Giuseppe to the base image that Bryan had picked for the Toolhub Docker containers. Giuseppe's bug fix removed an upgrade of the pip package manager software beyond the version provided to us from the upstream Debian package repository. This was a proper fix for the intended purpose of the python3-build-buster base image, but it had the side effect of leaving Toolhub with an older version of pip that did not support the manylinux2014 wheel format now being shipped by pyca/cryptography.
The quick fix for this was to pin our usage of python3-build-buster to an older version that still includes the pip upgrade. The long term fix is relatively simple though, and involves cleaning up some technical debt that was only documented with a comment reading FIXME: we need a non-dev Python3 buster base image in Toolhub's Blubber configuration file. This obscure comment has been promoted to a proper task to create a base image for Python 3.7 project work that is separate from the python3-build-buster base image that is used to build wheel files.
A new python3-buster base image is now available for Toolhub and other Wikimedia projects to use. Toolhub will be switching to it as soon as a related change to Blubber is merged to provide the same upgrade of pip that was previously included by error in python3-build-buster no matter what base image is used to apply a python variant configuration.
API for history and diff of toolinfo records
[edit]We have merged a patch adding API endpoints for history and diff. The API now exposes these endpoints:
- GET /api/tools/{name}/revisions/ - List all revisions of the given record
- GET /api/tools/{name}/revisions/{id}/ - Get revision details
- GET /api/tools/{tool_name}/revisions/{id}/diff/{other_id}/ - Get a structured diff between two revisions
The diff endpoint returns a JSON Patch description of how to transform the {id} revision into the {other_id} revision.
After a bit of discussion this week between Bryan and Srishti, Bryan will be spending some additional time to design API endpoints to make some undo/revert patrolling actions possible via a single API call rather than forcing all clients to implement these activities using local business logic that builds on the primitive operations of requesting diffs and posting edits.
Direct tool registration
[edit]Srishti has started work on the UI components and workflow for creating a toolinfo record directly in Toolhub. This work builds on the PUT /api/tools/{name}/ API endpoint added a few weeks ago.
Discussion between Srishti and Bryan about questions that Srishti had about schemas has led to the creation of phab:T274679 to remind Bryan to start working on documentation on how the Toolhub API and its OpenAPI spec are intended to be used as well as the difference between the schema implemented by the API and the schema for externally hosted toolinfo.json documents.
Small changes
[edit]- gerrit:662029 toolinfo: fix icon validation regex
- gerrit:662030 api-doc: add descriptions to CommonsFileOutputSerializer
- gerrit:662031 toolinfo: Reduce serializer config duplication
- phab:T274018 Change method used to populate create/modified timestamps
- gerrit:663591 fixtures: update demo data
Wrap up
[edit]The cryptography package issues ate up a bit of Bryan's time this week, but in the long run that work will make Toolhub's pre-deployment reviews a bit smoother.
Srishti will be taking some well deserved time off next week, so there will probably not be a large amount of progress on the toolinfo creation screens in next week's report.