Experimentation

Experiment is a stage in the development of changes to policies and processes in which an idea is being tried on a temporary basis, perhaps with a limited scope of application. Many current policies, such as limiting page creation to logged-in users, began ostensibly on an experimental basis. Proposed deletion began as an experiment; while discussion on the proposal progressed, some users simply began applying the new process on a test basis.

Considerations

Experiments can be useful for determining how an idea would work in practice. Debates on proposed policy changes often center around speculation or "what-if" scenarios whose significance and actual likelihood of occurring are difficult to assess. In some cases, an experiment can settle those issue with more finality and even reveal unanticipated benefits or drawbacks not mentioned in the debates. Before objecting to the launch of an experiment based on what might occur, opponents of a proposal may wish to consider whether their case might be strengthened by the results of the experiment if it were allowed to proceed. It may be more convincing to be able to say, "We already tried this, and the result was _____" than to say, "I think that if this proposal is adopted, the results will be _____."

In other cases, existing knowledge may make the results of an experiment a forgone conclusion; or the risks and costs involved may outweigh the potential benefits. As Tim Vickers said in reference to restoring anonymous users' rights to create new pages, "Sticking your finger into an electric shredder is not a sensible experiment, we know exactly what the result will be and confirming the prediction will be painful." Proponents who view a particular experiment as benign to nonparticipants may be surprised when the community vetoes it, and wonder, "Why do you care?" There are many possible reasons. In some cases, the community concludes that a proposed experiment would divert participants' time from more useful contributions, to the overall detriment of the encyclopedia. Accordingly, it acts preemptively to nip it in the bud.

Political considerations can also come into play. By default, discussions with no consensus result in no policy change. When that outcome appears likely, opponents may seek to block a suggested experiment as a way of thwarting the overall proposal. Bearing this in mind, proponents may wish to frame their pro-experiment arguments in terms of superordinate goals. Will this experiment provide information that could be useful in other debates, or benefit the encyclopedia in some other way? In the face of intractable opposition, supporters may wish to propose conducting an experiment in a more receptive venue, such as another WikiProject, or even another wiki. For instance, some proposals begin on the German Wikipedia and are then recommended for implementation on the English Wikipedia.

Steps

Planning the experiment

It is good to design an experiment in such a way that it is easy to administer. Certain experiments, such a new classification scheme or promotion process for articles within a WikiProject, require little cooperation from outsiders, and thus are relatively easy to arrange and coordinate. Others, such as experimental deletion systems, may need to be planned more thoroughly, since users from all over the encyclopedia will have to work around it. One may need to designate a certain test group that will be off-limits to normal processes for the duration of the experiment. If the test subjects can be arbitrary chosen, then the selection basis should be a criterion that can be easily and objectively determined, such as articles whose names start with the letter A. This reduces opportunities for selection bias and makes it easier for others (e.g. users watching New Pages Patrol and Recent Changes Patrol) to steer clear of the experimental subjects. It may be good to open up a proposed experiment for brainstorming to generate ideas on how it should work. Proposed experiments can be publicly announced at places like Village Pump Proposals or Metapub.

As with any experiment, a hypothesis should be made and methods should be proposed for testing it. How will the results of the experiment be measured? What baseline or control group will the results be compared against? What extraneous factors need to be guarded against?

Conducting the experiment

Supporters should be prepared to back up their idea with any necessary work. Templates may need to be created to provide adequate notice of what is going on. Some people may be unaware of the experiment and inadvertently interfere; or something unexpected may happen that makes the experiment go awry. These situations will need to be addressed quickly and effectively in order to ensure success within the limited time frame in which experiments take place. Failure to conduct a much-publicized experiment competently may harm one's prospects of getting support for a future experiment.

Reporting the results

Results should be documented both at the end of the experiment and during the experiment, if possible. This will help reveal problems with measurement or methodology and allow time for adjustment before the period of the experiment is over. It may be necessary to make changes mid-way in order to get it on a track that will produce useful results. Results of experiments should be summarized at Experiments/Results with appropriate wikilinks. This provides a central location at which people can review experimental results, for instance, if they are thinking a proposing their own experiment.

Avoiding disruption

Ideally, experiments should be planned and conducted in such a way as to minimize disruption to the larger community, while still producing valid and useful results. But sometimes, the fact that an experimental idea is untried can make the extent of resultant problems difficult to predict with certainty. That may be, in fact, one of the things that the experiment is intended to ascertain, and it can seem like a catch-22 when experimentation is prohibited on that basis.

The question of how and when to be bold in experimenting is a judgment call. Undoubtedly, enactment of some policies (such as proposed deletion) has been hastened by users taking the initiative to apply them while discussion was underway. In some cases, experiments can benefit the community by helping it make a more informed decision. Enacting an untried policy and then testing it – or taking the "safe" route and maintaining the status quo because of unfounded concerns about a beneficial proposal – can actually be more costly than simply conducting a test. On the other hand, experimental action, if perceived as presumptuous, can stir up opposition that hurts the proposal's chances.

Therefore, the best cases to be bold are when there is reason to believe that objections are unlikely. One will probably want to try a relatively limited application of a proposal at first. That way, it is more easily reverted and less likely to be viewed as offensively reckless or disruptive. Such cases are essentially an application of bold, revert, discuss. On the other hand, if objections are anticipated, it is probably better to seek approval from the community first, perhaps by posting at WP:RFC or the Village Pump. An experiment may draw attention to an idea, and may have to be advertised in order to attract sufficient participants, but making a point is not its primary purpose.

As experimental policies are not considered official, users apply them at their own responsibility and risk. Experimental behavior, being outside the norm, is more likely than most edits to attract suspicion or concern, especially if other indicators suggest that the behavior is not in good faith. Experimenters may do well to anticipate the heightened scrutiny, disclose their intentions clearly, pay close attention to the community's response, and be mindful of conclusions observers might draw from their unrelated actions.

See also

• en:Template:Experimental
• Policies and guidelines
• How to create policy