Training modules/Feedback/Preliminary surveys

Drafting/discussion spaces

Discuss design, accessibility, and formatting of modules

Previous research

Full surveys

Full report - Community
Full report - Academics
Full report - Industry professionals

Community responses

Full report - Community

The Support and Safety team conducted a survey on the topic of training modules, across four different key audiences: functionaries, non-functionaries, academics and industry professionals. It was launched on August 15, 2016, and was closed three weeks later. The functionaries and non-functionaries survey presented in this report attracted 32 respondents from functionaries on various Wikimedia projects—users with oversight and/or checkuser permissions, stewards, and members of active Arbitration Committees. A similar survey was sent out to non-functionaries who are involved with running Wikimedia events offline; of those questioned, eight responded.

Q1: What aspects of working on harassment cases are most challenging?

Responses to this open-ended question were grouped into rough categories. The content of each response could therefore fall into a number of categories. Among functionaries, the most common aspects named here included difficulties in dealing with users or the community (21.2%), identifying or defining harassment itself (14.9%), off-wiki harassment (12.8%), and technical limitations or a lack of effective deterrents (12.8%).

Among the non-functionaries, common answers included providing support and advice to victims of harassment (21.4%), difficulties in dealing with users or the community (21.2%), identifying or defining harassment itself (14.9%), and preventing further abuse (12.8%).

Q2: In your opinion, what are the most useful existing on-wiki resources on this topic?

Many of the functionaries surveyed suggested they were not aware of any useful resources currently in existence on Wikipedia (63.3%). Of the respondents who did supply useful resources, five mentioned on-wiki policy, two mentioned on-wiki technical tools, and another two mentioned Wikimedia Foundation support.

Of the non-functionaries, two said no useful resources existed, but three quoted on-wiki policy and another put forward an initiative undertaken by members of the community offwiki.

Q3: Have you taken part in any online training on handling harassment in the past?

The large majority of functionary respondents (81.3%) had not participated in online training on handling harassment in the past, though the figure for non-functionaries was lower, at 62.5%.

Of those that had taken part in training previously, most of the functionaries did not find it valuable; the average star rating among the sample group was 2.4 out of 5. However, the non-functionary group (which was a much smaller sample size) found considerably more value in theirs, where the average was 3.67.

Upon request, four sources of training were explained or linked to—two from each group. None was Wikimedia-provided. One from each group was employer-provided, while the two remaining were from other sources.

Q4: Have you been offered advice or training materials by your team or colleagues on how to deal with harassment?

Of the 31 functionaries responding to this question, 23 (74.2%) stated that they had not been offered advice or training materials from their colleagues on dealing with harassment. This suggests there is a lack of institutional knowledge in this area. Of the non-functionaries, five of the eight (62.5%) stated they had received advice or training of this variety.

Those responding "yes" also stated where this training was located. Altogether, 36.4% of the training materials were privately held, 27.3% was in-person training, and 36.4% was online (with half of the online materials being on-wiki).

Q5: Using a scale of 1-5, how important are the following techniques in your work?

Both groups of survey respondents were asked to rate out of five the importance of certain techniques in their work: De-escalation (reducing negative interactions between two or more users) Dispute resolution (structured attempt to resolve the underlying dispute) Enforcement (imposing penalties on users)

With five being the most important, functionaries overall stated "enforcement" the most important with an average of 3.69 stars. Second was "de-escalation" at a 3.41 star average, with "dispute resolution" at 3.31 stars.

Non-functionaries strongly favoured "de-escalation" (4.75 stars), with "dispute resolution" in second with 4.25 stars and "enforcement" some way behind with 3.13.

Q6: Using a scale of 1-5, how much difficulty do you and your team have with the following issues?

Users were asked to rate the following issues out of five in terms of the difficulties each pose to their work:

Learning to use technical tools (e.g. filters, global locks)
Users trying to “game the system” or file false allegations against others
Defining what is and is not actionable harassment

Perhaps unsurprisingly, few respondents reported difficulties in learning how to use technical tools (2.07 stars out of five for functionaries, 2.38 stars for non-functionaries). For the surveyed functionaries, the most difficulties were reported with users trying to game the system (3.61), with defining actionable harassment a close second (3.46). Non-functionaries had this the other way around (3.63 for defining, and 3.13 for users trying to game the system).

An "other" field was provided for users to report additional issues (or issues that were not mentioned). Nine functionaries gave responses for this. The most common were:

Dealing with offwiki harassment
Team communication
Lack of consistent policies across wikis
Repeat offenders and long-term abuse

Of the non-functionaries that responded, three provided an "other". They found difficulties with offline harassment, as well as attacks on dispute mediators, and issues with dispute resolution participants only commenting and not fully understanding the issues in play.

Q7: Have you ever been involved with helping to manage events off-wiki?

Answering "no" to this question meant skipping questions 8–11. Of the non-functionaires still in the survey at this point, all eight had been involved with helping to manage events off-wiki in the past. 18 of the 29 responding functionaries (62.1%) did the same.

Q8: Have you ever been offered training on how to handle behavior issues at events (or have you ever offered this to others)?

Only three functionaries (16.7%) reported having been offered training on how to handle behavior issues at events, or having offered this to others. Non-functionaries were split down the middle in their responses, with four responding positively.

Of those that had, all provided a description or a source for these trainings. Overall, a third of the sources mentioned were Wikimedia affiliates (Wikimedia Sverige and Wikimedia New York City), while others were from external groups like the Ada Initiative (mentioned by five respondents). All of this training was offline; not one respondent said their training was online.

Q9: Do you believe you would benefit from in-person training as well as online? Most of those responding to this question did believe they would benefit from in-person training as well as online, though the feeling was stronger among non-functionaries (87.5%) than functionaries (72.2%).

Q10: Using a scale of 1-5, how important do you feel it is for the following groups to receive training in how to deal with conduct issues at events? Respondents were asked to rate out of five how important they feel it is for the following groups to receive training in how to deal with conduct issues at events:

Event organizers
Event volunteers
Attendees

The 18 functionaries who responded to this question seemed to consider it most important for event organizers to receive training, with an average of 3.89 stars out of five. In second was event volunteers (3.33), with attendees a distant third (2.78).

Seven non-functionaries responded to this question; their choices ranked the same way. They seem to agree that event organizer training (4.43 stars) would be more important than event volunteer (3.86) or attendee (3.29). Their ratings, however, were on average 1.25x higher than those provided by the functionaries group.

Of the "others", functionaries suggested media handlers and partner institutions, while one of the non-functionaries suggested event support staff.

Q11: Are you aware of any best practices for handling in-person incidents of harassment?

Of the 18 functionaries who responded to this question, only eight (44.4%) were aware of any best practices for handling in-person incidents of harassment. Of the six sets of best practices provided by respondents, two were Wikimedia-related (either hosted on the projects or at Wikimedia events) and two were from their workplace. Others referenced other conferences' best practices.

The links provided by respondents included:

Friendly space policy (Wikimedia Foundation)
Safe space (Wikia - Geekfeminism)
Who is harmed by a "Real Names" policy (Wikia - Geekfeminism)
Code of conduct (Wikia - Geekfeminism)
Conference anti-harassment policy (Wikia - Geekfeminism)
Codes of conduct 101 - faq (Ashe Dryden)
CoC Pledge (Wikia - Geekfeminism)
Citizencodeofconduct.org

Six non-functionaries responded; their resources were from work (2), Wikimedia (1) and other (3). Links included:

"Clear, well-known contact with professional training in dealing with conflicts that are equipped with the house rules, check whether an immediate sanction required (in the worst case exclusion from one event) is present. If this is not the case, they help with a rule determination on site in order to avoid further cases of harassment." "It is important to have community leaders and arbitrators who understand and support anti-harassment policies and practices." "Develop ethics standards, develop code of ethics, implement it, appoint an Ethics Officer, preliminary fact finding, full investigation, make decisions, notify the parties of the decisions, run a post mortem, and so on."

Q12: How can we make an online training module interesting and enjoyable for users?

This question was posed as a free text field, and responses were grouped into broad categories as with question 1. 22 functionaries gave a response to this question, with their responses occasionally falling into more than one of the broad categories. The most popular themes raised were realistic examples or scenario-based training (20.8% of respondents), short, concise, and clear training (16.7%) and that making training interesting and enjoyable was "not possible" (12.5%).

Non-functionaries had different ideas, and their responses tended to be more varied. Two of the five responses mentioned realistic examples or scenario-based training (28.6%), while the figure was one each for the following:

Short, concise, clear
Not possible
Practical advice, "must be useful" etc
Video
Relevant to Wikimedia

Q13: Where should training on this topic be hosted?

In this section, respondents were asked to select only one option from a preselected list:

Meta
Local projects
External website
New Wikimedia wiki - e.g. "Training wiki"
Other

Among the 28 functionaries surveyed, almost half (46.4%) favoured training hosted on Meta. 32.1% suggested local projects would be a better fit, while only one (3.57%) preferred an entirely new wiki. None wanted an external website. Of the five "others", most had no opinion, reiterated that they did not believe training to be possible, only wanted it to be easy to find, and one suggested the content should be drafted first before being hosted somewhere with the necessary features.

All eight non-functionaries gave their opinion here. Three supported creating an entirely new wiki (37.5%) and a further three supported hosting the the training on local projects (37.5%). Hosting the training on meta, and on an external website, earned one vote each (12.5%).

Q14: Which of the following features do you prefer for presenting training materials?

Here, respondents were asked to select as many of five options as they felt appropriate:

Video
Audio
Text concise enough for one page
“Tutorial” or "how-to" style
Interactive, with activities or quizzes
Longform text
Other

Of the 28 functionaries responding to this section, 19 of them (67.9%) preferred a "tutorial" or "how-to" style for the training materials. The options "text concise enough for one page" and "interactive, with activities or quizzes" were selected by thirteen respondents (46.4%). Also popular were video (39.3%) and longform text (35.7%). The one useful "other" was to include infographics.

The eight non-functionaries seemed to have different preferences. All but one of them expressed a preference for video training (87.5%), though also popular were "tutorial" or "how-to" style (62.5%), interactive with activities or quizzes (62.5%), text concise enough for one page (50%), and audio (50%).

Academics responses

The survey of academics presented in this report attracted 37 responses from researchers and academics active in the field.

Full report - Academics

Q1: Are you aware of training that has mitigated online harassment in any online community or improved the preparedness of volunteers to assist others in dealing with online harassment?

Of the 37 academics who responded to this survey, the vast majority of them (91.89%) said they were not aware of any training that actually mitigated online harassment in any online community or improved the preparedness of volunteers to assist others in dealing with online harassment.

Of those that were successful, they suggested that peer pressure and active leadership within the communities was a key factor; "active leadership from the top".

Q2: How would you measure success of an online training project focusing on mitigating online harassment and/or assisting victims of online harassment?

This question received 18 responses comprising open text answers, which were roughly categorised based on their content. Seven of these responses (38.9%) cited a reduction in incidences of harassment as a major factor in measuring success of an online training project about this. Other popular measures of success included bettering the community's recognition of harassment (22.2%) and regular surveys of harassment victims and/or the functionaries dealing with this harassment (22.2%).

22.2% of the respondents stated that they had no answer to this question, or that there were no effective methods of measuring success. Others suggested a training project would be successful if it led to more inclusion, more space and methods to counter harassment, or more victims reporting harassment.

Q3: What factors do you feel we should keep in mind in teaching people how to differentiate between harassment and legitimate debate, particularly on controversial topics?

This question received 18 responses comprising open text answers, which were also roughly categorised based on their content. The most popular response was learning to recognise ad hominem attacks, which several respondents said do not form part of rational debate (38.9%). Another popular sentiment was that not confusing disagreements with actual harassment was the most important factor here (33.3%). 16.7% of those responding explicitly said that maintaining participants' rights to free speech was important for healthy debate, and should not be stifled.

Other responses included: Bearing in mind the power of individuals involved (11.1%); vocabulary/choice of words (11.1%); speaking to those who have experienced harassment (5.6%); "emotional trigger warnings" (5.6%); showing examples of both harassment and rational debate for comparison (5.6%); avoiding systemic, racial and gender biases (5.6%).

Q4: Are you aware of research on the best ways for online communities to: (Please select all that apply and share paper titles or links, if possible.)

Respondents were asked to indicate if they were aware of research in the following topic areas, listed here by the number of respondents replying "yes":

Create and deploy policies that prevent harassment within communities (5)
Raise awareness and understanding of the realities of harassment, in relation to cultural values or differences (3)
Recognise and reach consensus on an operative definition of “harassment” (3)
Handle, to the satisfaction of the largest number of involved parties, individual types of harassment or individual harassment cases (2)

Q5: Are there any efforts at your institution or related entities, addressing conduct at in-person events and/or providing guidance on handling issues?

Of the 18 academics responding to this question, seven (36.4%) said they were aware of efforts at their institution addressing conduct at events or providing guidance on handling issues.

Four of the six (66.7%) who provided descriptions for these materials stated they were hosted by their university or other similar institution. Others said their efforts were either hosted online (16.7%) or didn't provide a link because they weren't effective (16.7%).

Q6: Do you know of any scholarly research on the topic of preventing/educating about conduct issues at conventions, conferences or events?

Only two respondents (10.5%) were aware of any scholarly research on the topic of preventing/educating about conduct issues at conventions, conferences or events. Resources linked included the British Philosophical Association, the Feminist Philosophers blog, and an article from Briarpatch Magazine on the harms of call-out culture.

Q7: What formats of optional training, regardless of topic, do you believe to be effective in volunteer-led or crowd-sourced environments?

19 of the academics gave their opinions about the most appropriate format for optional training in an environment like the Wikimedia communities. Of these, the results were:

Single-page of text — selected by 63.16%
Video — 57.89%
“Tutorial”- or how-to-style — 57.89%
Interactive, with activities or quizzes — 52.63%
"Gamified" training, possibly with incentives — 42.11%
Longform articles — 21.05%
Audio — 10.53%
Other — 5.26%

The one "other" believed that debated cases deemed problematic in public was a more effective strategy in this situation.

Industry professionals responses

A survey tailored to other online platforms generated six responses from industry professionals.

Full report - Industry

Q1: Does your platform offer anti-harassment training, such as how to prevent, or how to react to, harassment?

Of those surveyed, three (83.3%) were from platforms offering this training. When asked to describe the type of training available to them, we received the following 5 responses:

"We have Title IX training, diversity training, and LGBTQ safe space training, as well as opportunities to learn how to identify and manage specific behaviors within the classroom environment."
"Training manual, one on one discussions with administrators, policy guidelines for dealing with pretty tricky situations. Links to past situations."
"Instructor led for new hires and self paced interactive training for continuing education. All scenario based."
"We go to colleges and universities around Pakistan and deliver trainings to young girls about online harassment and cyber security. The sessions are interactive and cover local laws, the girls' rights, reporting mechanisms and prevention information."
"Multi-faceted."

While most of the training described to us was available only to paid staff, and not to volunteers (80%), all of it was mandatory for their staff. There was some availability variation according to experience. All four professionals responding considered their training to have produced positive outcomes or improvements around the issue of harassment:

"A number of girls have reported cases of harassment to us directly, during and after the training (which they were reluctant to approach the authorities with) and has resulted in them either reporting the cases or DRF taking the case further."
"I found the Title IX harassment training incredibly informative. I have learned how to speak with my peers about issues tied to harassment. I have learned the importance of educating certain vulnerable groups (graduate students, international students). I have learned how to listen and advocate for victims."
"We have only paid staff, so the training comes as a job requirement. The training primary outcome is to help eliminate subjectivity from agent workflows. There has to be a policy and a threshold of behavior that separates free speech from harassment, bullying, illegal solicitation of minors, terrorist radicalization... This threshold has to be documented, measured, and defended. You can't do that without 1: a documented standard 2: Consistent Training, 3: A quality assurance program that measures adherence to training and policy. Once you have all those things, you can use additional cycles to refine policy, identify edge cases…"

Q2: Are you aware of any training modules or coursework outside of your organisation that address the issue of online harassment?

Two of the four who answered this question (50%) said they were aware of any training modules or coursework outside of their organisation that address the issue of online harassment:

WMC Speech Project
"Most social media companies have this type of training."

Q3: Does your platform host or sponsor in-person events? Do you provide special training to the organisers of the events in handling code of conduct violations/harassment/sexual assaults/etc?

Of all four industry professionals who indicated that their platform hosts or sponsors in-person events, three responded positively when asked further whether their platform offers special training to the event organisers in handling code of conduct violations, incidents of harassment, sexual assault, and so on (75%).

One professional indicated that the training on offer was in-person only (33.3%), while two indicated it was both online as well as in-person (66.7%).

Two professionals responded when asked whether the training is conducted at or before the event commences. Of those, one indicated that the training takes place at the event (50%), while the other indicated that the training is offered both before as well as at the event (50%). One professional's training on offer was voluntary (50%), while the other was mandatory (50%). Both of these trainings offered for event staff or volunteers used self-sourced staff.

Q4: Are you aware of any training resources for event staff/volunteers on how to prevent or respond to harassment at events they attend or coordinate?

Three out of four professionals indicated that they are aware of training resources for staff and volunteers involved in running events, on prevention and response to harassment incidents.

Q5: Please rate the following forms of training based on your experience:

The four industry professionals still participating in the survey were asked to rate training formats in terms of effectiveness. Video content was seen as an effective option by three respondents (75%), no other format received a more than 25% indication of effectiveness.

The two professionals who submitted an alternative medium under "other" identified scenario-based training and discussion groups.

Q6: Do you have any advice for us, not addressed by the questions above, that might help us create training modules for our community?

This question received four open-ended responses. Some quotes from these responses:

"...Local attitudes towards women for instance can make harassment take on different and in some contexts, deadly forms, and those dangers must be addressed."
"Correct for bias in any data or algorithmic products. Survey users and use intersectional data analysis to identify relevant differences in people's experiences and perceptions. Learn about how speech dynamics are affected by identity, gender, power, etc. because these dynamics are diffused in online commentary and interactions."
"For the love of humanity, do not gamify this. People targeted for harassment have killed themselves over it. Gamifying any aspect of this topic is tasteless and undermines the severity of the issue."
"The key is to [develop] a thorough gap analysis between current state and where you want to end up. You have to measure current state first. Don't assume. Then structure the training to address the learning objectives... Balance carefully the need for cultural and regional diversity with the need to have a common user experience. Finally, it is not enough to make the right call or do the right thing, you have to explain it the right way."