Article validation proposals/Page 1

The Article validation proposals have been split into multiple pages to keep them at a manageable size.

A common theme among many suggested methods is to distinguish a public or stable version from the latest version of each article. (Contrast this with the implemented validation method for TomK's WikiReaders, which only enhanced the community visibility of selected articles)

Wiki version and stable version

Main article: en:Wikipedia:Stable versions proposal

There should be two versions.

The wiki version, which is editable by everyone anytime, and
The stable version, which may not be editable by anyone

Note: This means that if a particular version becomes flagged validated, new edits to it will not be marked validated, and there will be a seperation between latest and stable. (Perhaps minor edits can inherit the validated flag?)

The stable version is defined by validation of one wiki version; it is marked with a flag. The successive validated versions are visible in the history of the article.

We could add at the bottom of the article "This article has been validated by XX editors," or "This article has not yet been validated by any editors. See last validated version."

How to name/view (the last) stable version(s)

The last stable version is visible to a reader

in a new namespace, on wikipedia itself, this namespace containing non editable articles
or in another website (with cross-domain links)
or in place of the most-recent wikipedia revision (with a diff and the most recent revision shown on edit)
- Seeing the last stable version rather than the wiki-version might be chosen in the user preferences.

Validation via extra metadata

In preface: the suggestion below is a scalable technical change to make a variety of validation and reviewing efforts faster and more reliable. It is a medium-term change, and does not conflict with the excellent and immediate #Natural validation process suggested below.

About the final step: The final step in validation should be grunt-work : a user flagging a clearly valid article "validated". This should be human and not automatic. In the proposal below, this final step can be taken by any admin once certain conditions are met, but it could probably be carried out by any user (and undone only by admins).

The main problem is not in doing the actual validation -- FA works without restricting who can update that page -- but in distributing the work of steadily improving invalid articles, so that individual editors can specialize in what they do best.

Step 1

Allow our thousands of casual readers to inform the validation process, in a better-defined and more cleanly-parallel fashion than the current freeform method (talk pages). Anyone who can edit the wiki should be able to help validate (or unvalidate) an article. For example: give readers a review this article option which lets them review many distinct aspects of articles. Allow them to either set review flags relating to the article content, or flag the article as in serious trouble.
Review Flags, + | neutral | - (anything but neutral requiring explanatory comment):
1. factual accuracy, (the most time-consuming part of reviewing; requires checking with external sources and/or expertise)
2. quality of content,
3. neutrality of tone,
4. completeness of coverage (missing key info? too little info for a major topic?)
Unary flags: (these setting each of these also causes the article and any special pages to show appropriate templates and content)
1. Stub
2. Copyvio (requires comment)
3. Inappropriate content (requires comment)

(These might simply be determined by the inclusion (or not) of a category or a template)

This allows a reviewer to either validate or invalidate an article in four facets of quality, and to quickly flag pages as stubs/copyvios, or for VfD. Presenting a set of facets encourages editors to think along the lines of validation criteria, and using flags makes tracking/responding to reviews automatic.

Step 2

Allow active editors to quickly and cleanly respond to negative article reviews: let them disassociate old comments as they are addressed/become outdated, with a responding comment.
Allow an editor to see a detailed review history of reviews and comments

June 8, 2004 20:41:06

Fuzheado

A:+ (more detail than my Grove's has)

Q:= (OK) N:- (I tried to neutralise the bit about Iraqi civilians, but it still reads like propaganda. Some quotes from the other side would help balance out the "In the Media" section.) C:= ()

Technical thoughts:

Each review, and accompanying review notes (like edit summaries) are associated with a reviewing user, time, and article revision. They are also associated with each successive article revision unless they are deleted (by an admin, for vandalism) or responded to by another editor (in which case they are disassociated from the current revision).
Once an article has been given a thumbs-up in each facet by at least one reviewer, and all negative reviews have been responded to, an admin (after verifying that the responses were in good faith) can follow a "validate this article" link and brand that revision validated, creating a new revision (without altering the text). Someone else might come along the next minute and see that same text and give it a negative review... which would have to be addressed before the article was next validated.
Readers can look for the last {reviewed, validated} version, or see a 'list of all reviews', perhaps even a 'list of all validated versions'.
The current revision of an article is associated with the latest reviews of all users who have reviewed the article in the past - excepting reviews that have been responded to.

Possible metadata interface design

Interface design: associate a color and letter with each facet.

[A]ccuracy : Green
[Q]uality : Blue
[N]eutrality : LtRed
[C]ompleteness: Purple

Let as much information as you can trickle up to the readers, in a subtle fashion.

At the upper-right corner of each article there could be a tiny globe; grey by default, but colored in if that version is actively validated (blue?) or unvalidated (red?).
On mouseover, a grey globe might text-message "This article has not yet been reviewed. ", or "This article has not yet been validated", or "This revision has not yet been validated", right next to a "see last validated revision" link).
Note: most current revisions will be unreviewed; perhaps the globe can be suppressed in those cases, to minimize the impact of this system on typical browsing.
At the bottom of a validated revision would be a link to "validation details", which would be a page displaying a list of recent review comments and responses, along with a list of validating users (that is, users whose last review had been positive), with the dates (colored according to which facet they were reviewing?) and comments submitted with their validations.
Validated articles would have a "last validated revision" link near the globe at the top, which would take them to the appropriate revision.
At the bottom of each page would be a "show review history" link that would display a list of each reviewed version, with a review summary, something like this:

June 8, 2004 20:41:06 A Qx2 Fuzheado, 64.102.12.3 June 2, 2004 14:12:11 A Q Kpjas May 14, 2004 07:36:40 Ax4|1 Qx3 LittleBuddha, Fuzheado, Fuzheado, 143..., Anthere, Belizian, J-VH, Tomos May 11, 2004 17:50:01 Ax1|1 Q Tomos, 143...

Technical interface design

Make reviews (and responses to reviews) show up on watchlists and a special V-RC page. This will help draw attention to validated articles, to make sure that people don't mistakenly validate a poor article.
1. Publication teams looking for stable versions to copyedit and fact-check into a published work can start with this general validation system, refine it through contact with interested editors in the subject-area wikiprojects, and keep an eye on the reviewers/validators -- only team members should be responding to / unflagging reviews or validating articles.
2. Admins should be able to revert a mischievous review by vandrolls; admins could archive a mischievous review flag, so that it shows up when viewing the article revision (along with a note: "reverted on <date> by <admin>") but doesn't appear in the "list all validated versions" summary view, or in other review summaries.

I hope this was not so long that it now seems overcomplicated! The basic idea is simple. +sj+ 07:20, 23 Jun 2004 (UTC)

Weighting and averaging reviews

For most articles, this should be unnecessary. Every objection or negative review of an article should be responded to before it is validated (Most articles will perhaps suffer from too few reviewers, rather than an excess of active pessimists who are never satisfied). For those few cases where some objections cannot be met or cannot be agreed upon, there should be some automated rule of thumb to help an admin decide whether an article has support for validation or not.

Appropriate levels of consensus We may want to isolate those aspects of articles (accuracy) which demand near-unanimous support from those more subjective aspects (completeness) for which a majority is acceptable.

Playing the game Since we only need to worry about this for controversial articles, we should expect that some energetic people will try to subvert the reviewing system. Verifying factual correctness, in particular, is time-consuming, since a random well-meaning Wikipedian cannot tell simply by reading an article whether its facts are correct (whereas the other facets may be difficult to work out, but success is easier to recognize/validate) -- and, moreover, it is a delciate facet, one which we would like validated articles to have near-unanimous support.

Combining Openness and Trust' It is important for all contributors to be able to help review articles. It is also important to have a notion of trust -- via which the opinions of trusted contributors may be heard above any focused effort among untrusted contributors to subvert the review process.

Suggested method

Make it hard for bots to submit reviews, on general principle, while allowing one-visit wonders to leave productive input.
1. Use a broad trust metric to simplify the problem.
  Have a 'reviewer' flag for anyone who bothers to read what reviewing is all about, and apply it liberally. Let anyone review an article, but post reviews by non-reviewers at the end of the article talk page, instead of as normalized article metadata.
  
  Put a become a Wikipedia reviewer!" link at the top of the Review page for accounts that are not yet reviewers; require them to visit a page explaining the review process, and to submit a request to be a reviewer. Process requests from IPs manually; scan others after the fact. Allow admins to manually set/unset a user's "reviewer" flag, with minimal red tape.
2. A en:CAPTCHA test (aka "copy the letters from this slightly hard-to-read image into the textbox") can offer some basic protection.
For the purposes of averaging reviews, calculate separately the consensus among all users commenting on the talk page (untrusted reviews) and that among the reviewers.
- Reviews from accounts which have since lost their "reviewer" flag (for trolling, say) should be counted as untrusted reviews.
- Allow untrusted reviews to be ignored if they give no reason for their review

This allows requiring supermajorities among active reviewers without giving mischief-makers 9:1 leverage in the review system. For example: (note: neutrality is the facet most likely to be targeted by large special-interest groups, and the most difficult to demonstrate conclusively)

Level of consensus required (all users | reviewers):  
         [A]      [Q]      [N]      [C] 
       45%|90%  40%|80%  25%|70%  25%|50%

+sj+ 03:01, 18 Jul 2004 (UTC)

Validation by committee

Create a "Committee of Experts" on particular topics from among Wikipedia contributors: For example, Camemebert has shown expertise in classical music, and Viajero has shown expertise in opera. Put them in a music committee, chaired by TUF-KAT (all names are only examples). Similar committees, larger and smaller, would exist for various topics, such as biology, history, Lord of the Rings, etc. Committee membership can be flexible, with new people joining as they evidence expertise.
Ideally, these "experts" would have to prove a) commitment to the Wikipedia project and ideals, and b) some expertise in the field in their non-Wikipedia life. A professor of biology would probably be better suited to chair the Biology Committee than a high school kid preparing for his A levels in biology.
The Committees would have several tasks. The easy ones are:
- determining categorization principles for that subject,
- determining what is missing from the list of existing articles,
- posting this information and encouraging gap-filling,
- validating articles.
The validated, or approved, article is flagged and [moved to a special page]. Once that happens, only committee members may alter it. It can now be accessed through a new, "Validated articles" section.
The original article continues to be edited by the community. The Committee observes these changes and at regular intervals decides which are worthy of incorporation into the validated article. This will require that people on these committees be open to change and committed to the Wikipedia ideals.
Validated articles can be produced instantly in a print version. Regular articles are in a state of constant flux. (note that for this stability, it suffices to mark one of the article revisions as 'validated' +sj+)
An impartial Overseer institution would be created to ensure that the Committee of Experts does not overstep its bounds. (... how about The Community At Large? or is it turtles all the way down? +sj+)

Problematic points

Who decides the committee's composition, and the rules according to which it makes decisions?
The parallel existence of validated (but editable) and regular-wiki article could bring a lot of complexity to the system. A simpler solution is that a validated article is non-editable at all. (... like an old revision. +sj+) (...or an actual old revision.)
We must not endanger this collaboration either by dividing our contents into good/bad or by dividing our volunteers into editors/validators.
The validation committee are not laymen (and if they are, they soon won't be after validating many articles), but some articles need to be readable to laymen too. R3m0t 17:56, 21 Jan 2005 (UTC)

Validation by rating/voting

(changed 'vote' to 'rate' where appropriate to distinguish yes/no voting from rating on a scale from 1 to 6 +sj+)

Another option would be individual voting a la "This version has been validated by X editors".

Any registered user can vote that an article is useful (yes/no); any page with more than x users voting 'yes', can be considered validated.
Option 2: Any reg. user can rank an article on it's usefullness, giving a value ranging from 6 (extremly useful) to 1 (rubbish). Any Page rated by more than x users to a score of 3 or more can be considered validated.

NOOOOooooooo!!!! <imagines the pain and the suffering... the painful suffering... the insufferable pain...> +sj+

ROFL!!!!!!! Anthere 14:14, 23 Jun 2004 (UTC)

Could somebody please clarify what is so funny? The idea as whole bringing 'objective' criticism to it? From the discussion, I would reconsider this proposition, as using a range of values and allowing rating "by any user" is not good. However, consider the Slashdot moderation system. It works very well on how /. "validates" comments, and with some minor adoptions could possibly work as well for a wiki. Axel Kittenberger 17:28, 23 Jun 2004 (UTC)

Another positive effect could be an implied campaign araising between editors to create the most useful (scored) articles, could be an useful addition to encourage quality, not? Axel Kittenberger 17:41, 16 Jun 2004 (UTC)

I also dislike the voting idea, as that might be easily manipulated by sockpuppets, and doesn't really solve the problem, as those voting, even with good intentions, might not have all the facts or even the knowledge to vote correctly. Burgundavia 08:30, 18 Jun 2004 (UTC)

Natural validation process

Example of what was done with the WikiReader

Prerelease often and announce it everywhere (mailing lists...).
Find writers who are already active in the topic and make sure they know about the reader.
Promise the very active editors to be mentioned specially in the print edition (they will love to see their name in a book)

This provides many "validators".

This seems to me the right way to start, even if one of the above technical solutions is later implemented. It can work very well with focused subject-areas that are non-controversial, as the WikiReader progress has shown. The first steps for implementing this are:

Decide on a manageable set of topics (this grows with each iteration)
Decide on the rough set of articles within those topics (some will not exist)
Get a few active, knowledgeable people to divert some of their energies to this effort;
- Then get other people excited about reviewing and improving those articles, by broadcasting the effort and emphasizing the excellent final goal (for this it is important that the goal be attainable)

Optimizing before a problem crops up is a waste of time; only after many people are excited, must one start to worry about mischief. (However, while embarking on the above steps, we can simultaneously agree on which technical solution to employ eventually, and coders & designers can start working on that) +sj+

Trust metrics

One broad trust metric was mentioned above; trusting users with 20 edits not to be bots. This is a poor way to reach that conclusion; feel free to suggest a better one. Similar trust-related traits about a user which can be determined rather automatically:

an account is not always a bot
an account sometimes makes valid edits
an account been vouched for by some other account
a user has put +10 (+N) hours into an account

With some combination of the above, it is much easier to filter out vandalism and other counterproductive contributions.

Trust isn't really a binary, yes/no, human/bot quantity. A degree of trust, determined by # of validated contributions and meta-validation and endorsements by those already well-trusted would be more resistant to bots as well as provide a metric for the quality/weight of a review. This is particularly valid for qualifying 'expertness' for reviews of factual accuracy/completeness. However, it would be important to consider different levels of trust per category. Intangir 10:45, 6 Dec 2004 (UTC)

Foundation wiki approval proposal

These are some possible solutions:

Add a publish button to every page.
Anyone can edit the wiki, but only approved people can approve an edit, so it isn't live until checked by an authorised person.
When an authorised person goes to edit an article they get a warning that an unapproved edit is queued. They check that edit. If they approve of it, they edit that version of the article and put it live. If they don't approve, it is removed from the queue and they edit the previous version of the article instead.
Unapproved edits can not be seen until you log in. They will not show up in search engines.
A query that reports all top versions which are not by approved people could be run to provide an approval queue. Approved users could go to Special:Unapproved to view these and publish any they agreed with.

Only serve cached pages, and only let approved people purge the cache. Unapproved versions would only be viewable in the page history, so would not be found by search engines.

Display a notice (something like the fundraising notices?) indicating that anyone can edit it and providing a link to the other version (the two versions would be the last approved version and the last version). Brian j d 03:01, 2005 Feb 24 (UTC)

Viewing unapproved pages

Readers should never see an unofficial version. You should have to log in to see an unofficial version. Therefore, there would be 5 types of user:

Reader: Not logged in. Can read live pages, but not drafts. Can not edit.
Logged in user: Someone with an unapproved account. Can edit draft versions. Can read unapproved versions.
Approved: Has been approved by a sysop. Can edit any unprotected page. Can choose to publish any edit.
Sysop: Can edit protected pages. Can add logged in users to the "approved users" list.
Bureaucrat: Can create new sysops.

Possible problems

One reason the Foundation wiki is not open to editing is that it allows the full use of HTML. To keep it safe, unapproved users should not be allowed to use HTML. If they use HTML, this should be rendered as plain text until someone approves it.

Comments

I am for the separation of unapproved draft / public version.
As for draft editing, I prefer to edit on meta first. Recent translation of Quarto vol.1 many (over 5) should engage ... I'm not sure that it is good such many people have their accounts on WMF and that same persons will take part in the coming issue ... How about we prepare first drafts on meta and move them to the WMF site as final drafts and then work there and use feedback on meta (as same as the recent French version correction). For avoiding the confusing, after moving the initial draft on meta should be deleted (or we should wait in a week and it would be too long?). --Aphaia 15:58, 27 Sep 2004 (UTC)
I agree very much with the solution, in part. Wiki's strength comes in both simplicity and flexibility. Requiring users to have an account to edit/publish may be well suited in some applications of Wiki, while in others it may be hindering. With this in mind, I believe either having editing levels(from anonymous usage to locked, with some resolution) or downright page permissions with regards to who(in respect of Approved Users, Users, and Anonymous) can read, edit, or publish may be in order.
In terms of drafts and publishing, I will comment based on my situation. I am part of a company who's leaning toward a Wiki to maintain documentation of its products. We want our customers to freely be able to make changes and additions to the documentation while being able to approve the validity of its contents. We either need to 1) filter all changes made by non approved-users, as is proposed in the solution. Or 2) have an 'approved' icon or message appear on a page which has been given approval by the Sysop or approved-user. In which case the page would then lose its approval by being edited by a non approved-user. --Brian 00:03, Oct 2, 2004 (UTC)

This sounds good but I think anyone should be able to edit discussion pages and have their edits appear immediately - how is the HTML problem dealt with on other wikis? 210.50.104.156 08:01, 17 Dec 2004 (UTC)

Lets keep it simpler and more similar to other wikis - have "readers" (users who are not logged in) and logged in users (users who are logged in but who are not "approved") treated the same! Also, I suppose sysops will be able to remove users from the "approved" users list?

Maybe the easiest way would be to only serve cached pages, and only let approved people purge the cache. Unapproved versions would only be viewable in the page history, so would not be found by search engines.
We have a static link to the history and then static links to the various versions, so why wouldn't these be found by search enginges? Brian j d 10:18, 2005 Jan 24 (UTC)

The Foundation wiki is just a wiki where every page is protected, right? Shouldn't things be treated the same way as protected pages on other w:wikis (other w:Wikimedia Foundation wikis, anyway)? Brian j d 09:06, 2005 Jan 27 (UTC)

People can work on drafts before putting it live. (This was listed under advantages before I edited that section.)
They already can, by putting the drafts on Meta, but this is bad as described above. Brian j d 10:01, 2005 Jan 29 (UTC)

I copied the following from the Advantages section Brian j d 03:02, 2005 Feb 16 (UTC):
all this is true. The current situation was among other reasons selected for being easy. Developing a "approval" system required time and a developer motivated. If such a system was developped for another, I think I would support it. Anthere
Huh? None of what you said makes any sense! Brian j d 10:01, 2005 Jan 29 (UTC)

`June 8, 2004 20:41:06`	`A`	`Qx2`	Fuzheado, 64.102.12.3
`June 2, 2004 14:12:11`	`A`	`Q`	Kpjas
`May 14, 2004 07:36:40`	`Ax4\|1`	`Qx3`	LittleBuddha, Fuzheado, Fuzheado, 143..., Anthere, Belizian, J-VH, Tomos
`May 11, 2004 17:50:01`	`Ax1\|1`	`Q`	Tomos, 143...