Talk:Machine learning models

Sources of the information in the model cards

@HTriedman (WMF): Thank you for creating this useful resource. I'm wondering though what the information in the cards is based on specifically, and would suggest to link such sources in each case, or at least document them in the overview page.

For example, in Machine learning models/Production/German Wikipedia damaging edit you (or your bot) wrote that German Wikipedia uses the model as a service for facilitating efficient vandalism triage, edit reviews, or newcomer support. But this model doesn't seem to be available in the German Wikipedia's RecentChanges filters (in contrast, say, to the the English Wikipedia).

Also, the card says that German Wikipedia decided to use this model. Over time, the model has been validated through use in the community. But at least on a quick search, I wasn't able to find a community discussion to that effect. If it exists, it would be valuable to link it from the card.

Regards, HaeB (talk) 05:15, 13 November 2023 (UTC)

Hi @HaeB: Thanks for these comments; they're good points. Right now for the purposes of ease of use, the model cards for ORES are created with a model-class-specific template (i.e. good faith, damaging, article topic, etc.). This was an effort at having relatively complete language coverage without having to write ~100 custom model cards, particularly with regard to publishing test statistics on a language-by-language basis. Unfortunately, because we started with English, this also means there's inaccuracies/difficulties in linking to proper documentation. On the other hand, language-specific ORES models are rapidly becoming deprecated (in tools like Automoderator) in favor of language-agnostic models on Lift Wing, so these cards will become archives of previous models relatively soon.

Regardless, your point below about making model documentation more visible at the point of use/prediction is very well taken. As Chris said, we're actively working on that! Anyways, thanks again. HTriedman (WMF) (talk) 18:46, 13 November 2023 (UTC)

Low usage

This page [1] and the individual cards (e.g. [2][3]) appear to attract very little traffic currently, indicating that they are not yet fulfilling their intended purpose of serving as the main point of community governance for WMF-hosted machine learning models as part of work to make open source, transparent, human-centered machine learning. Are there any plans to improve that?

For example, a recent changes patroller on enwiki who clicks the "How do these work?" link for the edit filters here arrives at the documentation page mw:Help:New filters for edit review/Quality and Intent Filters, which is quite informative but makes no mention of model cards.

Regards, HaeB (talk) 05:27, 13 November 2023 (UTC)

Hey HaeB! Yeah I think you are right. I'll create a ticket for starting to update some links and docs to direct people to the model cards. CAlbon (WMF) (talk) 16:01, 13 November 2023 (UTC)

Thanks Chris! Is there a link to that ticket? Regards, HaeB (talk) 03:13, 5 December 2023 (UTC)

Newly created! https://phabricator.wikimedia.org/T353025. Definitely put any additional thoughts in there. I think it is a larger problem than just a few extra links inside documentation. People who experience programmatic content should 1) know it is programmatic and 2) be able to click through to a model card that explains to then details about how the programmatic content was created. CAlbon (WMF) (talk) 21:39, 7 December 2023 (UTC)