Machine learning models/Production/Russian Wikipedia article quality

Model card
This page is an on-wiki machine learning model card.
A model card is a document about a machine learning model that seeks to answer basic questions about the model.
Model Information Hub
Model creator(s)	Aaron Halfaker (User:EpochFail) and Amir Sarabadani
Model owner(s)	WMF Machine Learning Team (ml@wikimediafoundation.org)
Model interface	Ores homepage
Code	ORES Github, ORES training data, and ORES model binaries
Uses PII	No
In production?	Yes
Which projects?	Russian Wikipedia
This model uses data about a revision to predict the likelihood that the article is of a certain quality level.

Motivation

This model card describes a model for predicting the quality of Wikipedia articles. It uses structural features extracted from the article to label Wikipedia articles with a probability score for each article quality class.

Wikipedia articles range in quality from rich, well-illustrated, fully-referenced articles that fully cover their topic and are easy to read to single sentence stubs that define the topic of the article but do not offer much more information. It is very useful to be able to reliably distinguish between these extremes and the various stages of quality along this spectrum. Wikipedia editors have developed rich rubrics for how to evaluate the quality of Wikipedia articles and are constantly assessing article quality to assist in coordinating work on the wikis. Editors use these quality scores to evaluate and prioritize their work. Researchers use these quality scores to understand content dynamics. Developers use these quality scores as filters when building recommender systems or other tools.

Wikipedia is always changing, which makes it time-consuming (and largely impossible) for editors to keep these quality assessments complete and up-to-date. An automatic quality model can help fill these gaps by evaluating the quality for articles that are unassessed or have changed substantially since they were last assessed. In doing so, it can provide researchers and tool developers with more consistent data and even potentially help editors identify articles that would benefit from a human assessment.

Users and uses

Use this model for

• high-level analyses of article quality trends
• filtering / ranking articles in tools – e.g. only show low-quality articles in a recommender system
• identifying potential ways to improve articles – e.g. using the lowest-value feature from the model as a recommendation

Don't use this model for

• projects outside of Russian Wikipedia
• namespaces outside of 0, disambiguation pages, and redirects
• directly comparing article quality across language editions – the scores are for a given project so e.g., an article that received a 0.5 score on English Wikipedia would get a much higher score if it had been on Simple English Wikipedia instead (because high-quality articles on English Wikipedia generally have more content than high-quality articles on Simple English Wikipedia)

Current uses

This model is a part of ORES, and generally accessible via API. It is used for high-level analysis of Wikipedia, platform research, and other on-wiki tasks.

Example API call:

https://ores.wikimedia.org/v3/scores/ruwiki/1234/articlequality

Ethical considerations, caveats, and recommendations

• The source data for this model is several years old — data drift may skew current outputs relative to the training data.
• The model does not currently take into account the quality of the specific writing, so a long article with many fake words would register as high quality. It does take into account structure though, so a long article would be penalized if it did not have many sections or was poorly referenced.
• Different wikis have different labeling schemes — do not use this model in conjunction with other models to conduct an interwiki analysis.

Model

Performance

Test data confusion matrix:

Label	n	~IV	~III	~II	~I	~ХС	~ДС	~ИС
IV	1465	1078	291	55	6	4	26	5
III	1465	402	625	275	28	15	108	12
II	1461	97	286	511	154	119	257	37
I	1458	51	104	252	247	357	273	174
ХС	1469	11	15	33	119	913	114	264
ДС	1488	2	59	55	34	110	1215	13
ИС	1476	13	18	15	51	280	55	1044

Test data sample rates:

	IV	III	II	I	ХС	ДС	ИС
sample	0.142	0.142	0.142	0.142	0.143	0.145	0.144
population	0.487	0.363	0.093	0.029	0.011	0.009	0.007

Test data performance:

Statistic	IV	III	II	I	ХС	ДС	ИС
match_rate	0.392	0.211	0.103	0.048	0.106	0.101	0.062
filter_rate	0.608	0.789	0.897	0.952	0.894	0.899	0.938
recall	0.736	0.427	0.35	0.169	0.622	0.817	0.707
precision	0.915	0.735	0.316	0.102	0.067	0.075	0.085
f1	0.816	0.54	0.332	0.128	0.12	0.137	0.152
accuracy	0.838	0.736	0.869	0.933	0.896	0.904	0.941
fpr	0.065	0.088	0.078	0.044	0.1	0.095	0.057
roc_auc	0.941	0.854	0.792	0.737	0.897	0.941	0.937
pr_auc	0.924	0.736	0.266	0.088	0.085	0.127	0.272

Implementation

Model architecture

{
    "type": "GradientBoosting",
    "params": {
        "random_state": null,
        "presort": "deprecated",
        "ccp_alpha": 0.0,
        "population_rates": null,
        "criterion": "friedman_mse",
        "max_depth": 5,
        "min_impurity_decrease": 0.0,
        "tol": 0.0001,
        "max_features": "log2",
        "multilabel": false,
        "min_samples_split": 2,
        "loss": "deviance",
        "init": null,
        "warm_start": false,
        "validation_fraction": 0.1,
        "labels": [
            "IV",
            "III",
            "II",
            "I",
            "\u0425\u0421",
            "\u0414\u0421",
            "\u0418\u0421"
        ],
        "min_weight_fraction_leaf": 0.0,
        "verbose": 0,
        "n_estimators": 300,
        "min_samples_leaf": 1,
        "max_leaf_nodes": null,
        "subsample": 1.0,
        "scale": true,
        "center": true,
        "learning_rate": 0.01,
        "label_weights": null,
        "n_iter_no_change": null,
        "min_impurity_split": null
    }
}

Output schema

{
    "properties": {
        "probability": {
            "properties": {
                "IV": {
                    "type": "number"
                },
                "III": {
                    "type": "number"
                },
                "II": {
                    "type": "number"
                },
                "\u0414\u0421": {
                    "type": "number"
                },
                "I": {
                    "type": "number"
                },
                "\u0425\u0421": {
                    "type": "number"
                },
                "\u0418\u0421": {
                    "type": "number"
                }
            },
            "description": "A mapping of probabilities onto each of the potential output labels",
            "type": "object"
        },
        "prediction": {
            "description": "The most likely label predicted by the estimator",
            "type": "string"
        }
    },
    "type": "object",
    "title": "Scikit learn-based classifier score with probability"
}

Example input and output

Input:

https://ores.wikimedia.org/v3/scores/ruwiki/1234/articlequality

Output:

{
    "ruwiki": {
        "models": {
            "articlequality": {
                "version": "0.8.0"
            }
        },
        "scores": {
            "1234": {
                "articlequality": {
                    "score": {
                        "prediction": "IV",
                        "probability": {
                            "I": 0.07871492795062543,
                            "II": 0.08158342781162584,
                            "III": 0.22262223416510799,
                            "IV": 0.5613913043883748,
                            "\u0414\u0421": 0.010626272568025808,
                            "\u0418\u0421": 0.02418415449291141,
                            "\u0425\u0421": 0.020877678623328844
                        }
                    }
                }
            }
        }
    }
}

Data

Data pipeline

Labels were collected from on-wiki judgements of article quality, and then joined with revision features to create a source dataset.

Training data

Train data was automatically split off from test data using functionality from the revscoring repository.

Test data

Test data was automatically and randomly split off from train data using functionality from the revscoring repository and held out during the training process. The model then makes a prediction on that data, which is compared to the underlying ground truth to calculate performance statistics.

Licenses

• Code: MIT license
• Model: MIT license

Citation

Cite this model card as:

@misc{
  Triedman_Bazira_2023_Russian_Wikipedia_article_quality,
  title={ Russian Wikipedia article quality model card },
  author={ Triedman, Harold and Bazira, Kevin },
  year={ 2023 },
  url={ https://meta.wikimedia.org/wiki/Machine_learning_models/Production/Russian_Wikipedia_article_quality }
}