Learning patterns/Being a judge
.jpg)
What problem does this solve?
[edit]Being a jury member may make you feel honoured but remember that this can also raise serious questions. The way to avoid jury members from being questioned has traditionally been to declare that the jury is unquestionable and by hiding the underlying discussions and declaring the result as final. None of this works entirely in the Wikimedia world because of our quest for egalitarianism, fairness, and transparency. Jury members should therefore put in a lot more thought into the processes that they take part in.
What is the solution?
[edit]Judging is finally an art and various situations can force you to take fairly arbitrary decisions. It can be unfair and the only way to making it better can be by clearly discussing the pitfalls and making sure that you are being sufficiently diligent. The final results can come as a surprise, and you may find yourself having views that conflict with those of others. The only way to improve the situation is to give it sufficient thought, and it is best to be involved in any exercise from the beginning so that corrections can be made well in advance. The more the discussion, the better the outcomes.
- Think about what the larger cause of the exercise is - why are you selecting people, or photographs, applications, or whatever else that is being judged - is it actually helping the larger cause? Are we hurting the larger cause in any way?
- Can all the jury members communicate in a common language?
- Have all the jury members taken part or are they in approval of the discussions of the aims, the selection criteria, and the process?
- Has there been an open discussion of the selection process in which the community and the potential selectees have had a say?
- Do you think you have been offered the jury position in a fair and transparent way?
- Is there a place where you can check if more suitable members have been asked?
- Has there been a place for community members to question the selection of the jury members?
- Consider if there a value for the jury members to specify their personal qualities, their biases, their limitations, and their approaches before hand.
- Is any third-party software being used for the judgement process?
- Has the software been tested sufficiently?
- Does it suit your purpose?
- Do judges need to communicate their thoughts on the entities being selected ?
- Should the software maintain independent judgement by the jury members and prevent one jury member from viewing the judgement of the other?
- Does the software allow communication between the judges?
- Does the software allow for changing judgements?
- Have you discussed how consensus is arrived on?
- Do you have a discussion before finalizing conclusions where conflicting judgements can be seen and re-examined?
Things to consider
[edit]- If jury members are experts on applying specific criteria out of several that have been set out, then do not average the rankings or scores of the jury members - they are judging along different dimensions. Allow them to discuss and come to an overall consensus.
When to use
[edit]- When you are invited to be a jury member.
- When you are designing an event where selection by a committee is involved.