Wikimedia Clinics/Retrospective

Abstract

This document offers a reflective evaluation of the Wikimedia Clinic program, after four months of operation as a pilot, and 11 Clinic sessions. It is based on solicited feedback and team reflection to reduce the burden on clinic participants by routine participant survey.

Context

The COVID-19 pandemic significantly disrupted the capacity-building plans from our team. With scheduled and planned in-person events cancelled, we recognized that there was a growing need to develop a space for the community to come together. Additionally, the lives of volunteers have been greatly impacted by the abrupt changes due to the pandemic and it was imperative that our team provide a space where informally connecting with one another was prioritized. Accordingly, we looked for a way to stay in touch with volunteers and to offer a channel where volunteers could ask questions about any Wikimedia-related topic, using live video calls as the closest substitute to in-person meetings.

We called the calls "Wikimedia Clinics", borrowing the metaphor from "legal clinics", in the sense of making expertise available for ad-hoc consultation on any and all topics.

Design considerations

We designed this program with these requirements in mind:

• make participation easy: we wanted people to be able to easily attend without needing to register or apply. Calls are publicized in advance, and reminders are sent in broadcast on multiple channels, but no registration is required to participate.
• be of service: we wanted the calls to be useful to volunteers, so we encouraged volunteers to ask any and all questions they may have, rather than insisting on a particular focus matching our specific areas of expertise. Consequently, we anticipated serving a kind of 'switchboard' function, helping questions reach the people at WMF most qualified to respond to them, freeing the volunteers from the (sometimes truly challenging) burden of figuring out on their own the best person to approach at the Foundation. We offered a commitment to get volunteers answers, even if not in real-time on the call. This meant doing internal follow-up after calls.
• accommodate volunteers, inclusively: we wanted to ensure the calls were held at times that were convenient for volunteers, not just participating staff. While it is impossible to make every call accommodate every volunteer around the world, we strove to schedule calls in an approximate late-afternoon/evening for the largest amount of volunteers, as well as scheduled some calls in evenings of other time zones, and on a weekend.
• create lasting value beyond the call: however useful the calls are, only relatively few volunteers would attend any one call. We wanted some portion of the substance of each call to be available beyond the call, so we committed to creating digests of each call. The digests are not full minutes, but rather concise and paraphrased summaries of substantive comments, questions, answers, and resources shared during the call. The digests are posted publicly on Meta. They serve: 1. People who could not make the call; 2. People who don't speak enough English but can get value from machine-translating a written digest; 3. People who did attend the call and remember some useful tool/link was mentioned, and want to find it again.
• safety: the calls should be a safe space, not only in the sense of the Friendly Space Policy, but also in not being recorded and posted publicly in posterity. Knowing the calls were recorded and put in the public record on Commons would have, we believe, caused a w:chilling effect on people's willingness to share what's on their minds and ask questions or request feedback.

Method

Outreach

The calls were announced on the common public communications channels (Wikimedia-l mailing list, Wikimedia General Telegram group, Wikipedia Weekly Facebook group, regional network lists, groups, and channels), to reach as many community members as possible.

One step we did not employ is posting on each community's Village Pump page; we did not have the resources to translate the invitations each time, and considered this less important than other translation requests from our volunteer translators, but also didn't want to alienate communities by not posting in the local language.

After the initial announcement in each channel, one reminder was sent on the day of the call.

Facilitation

Every call was guaranteed to have at least two WMF staff attending. Having staff attend was crucial to the value offered by the calls, as volunteers could convene their own calls whenever they preferred.

The calls were held under the Foundation's Friendly Space Policy.

One staff member was the lead facilitator of the call, introducing the conduct expectations, managing the speaker queue, ensuring people are heard and questions are addressed. Facilitation also helped comprehension of questions and answers through paraphrasing and checking with the speaker that the paraphrasing captured the substance adequately.

After a volunteer brought up a question, or shared something and requested feedback, the facilitator invited other attendees to respond offering answers or feedback, and after volunteers had spoken, staff would contribute as well, to the best of their ability. If staff were not able to offer substantive answers or feedback, they would note down the need and leave it for follow-up after the call.

To facilitate calls in languages other than English, the Community Development team had to rely on the kindness of colleagues outside the team (our thanks to them for putting in the time!). This is an unbudgeted resource we were utilizing.

Note-taking and digests

Notes were taken by staff (with occasional help from volunteers) on a live Etherpad document shared with the call attendees. Those rough notes, with the inevitable omissions and non-sequiturs, were processed – edited for conciseness and clarity, incidental details redacted – and augmented – improved with links to resources, tools, and policies mentioned, yielding digests of the calls.

In addition to a clear and concise summary of the substance of the call, digests also included follow-ups with any additional information necessitated by questions or requests made during the call.

The digests were posted on Meta, and links to them were shared after each call on the public channels used for announcing calls.

Speakers

Based on feedback collected in the early calls, starting with Wikimedia Clinic #007, each call featured a scheduled segment, where a designated speaker presented a pre-announced topic for 10 to 15 minutes, and took questions afterwards, before proceeding to the unprogrammed "clinic" part of the call.

Attendance and reception

Attendance of the Wikimedia Clinic calls was fairly regular: calls were attended by an average of 16 people and a median of 16.5 people, or an average of 15.3 people and median of 11 leaving out staff attendants.

The most-attended call, by a wide margin, was Wikimedia Clinic #009, whose pre-announced scheduled speaker was Denny Vrandečić introducing Abstract Wikipedia. It was attended by 52 volunteers and 6 staff.

Feedback

Each call ended with a request for feedback about the specific call and the program in general. Overall, feedback was positive and appreciated the opportunity for direct live contact with WMF staff. Early calls expressed an interest in having at least some topics planned and announced in advance. The different time zones and days of the week were also appreciated. Some people expressed specific appreciation for the public digests. When asked about Clinic calls in languages other than English (a possibility envisioned by staff to begin with), volunteers expressed a strong interest in offering clinics in other languages.

Based on feedback from the early calls, Clinics began featuring a pre-announced speaker and topic starting with Clinic #007.

Calls in languages other than English

From the first, the Community Development team considered it important to extend the opportunity of the Wikimedia Clinics beyond the English-speaking volunteers of our very diverse communities. While it is obviously impossible to offer the clinics in every language we have active communities in, we did want to go beyond English, based on language proficiency of staff members who were willing to help co-facilitate Clinic calls.

To date, we had one Clinic held entirely in Spanish (Clinic #006), and one in Russian (Clinic #011). Both of these are the languages of large communities, as well as intelligible for many additional volunteers, beyond the Spanish and Russian communities themselves, who understand them better than they do English.

The calls were attended by a similar amount of volunteers as the English calls, though, as expected, there was almost no overlap between the English-language calls and these non-English calls.

Topics in non-English calls

It would have been interesting to compare the topics that volunteers brought up in the two calls not held in English, to the ones brought up by volunteers in the English language calls. We only had two calls in other languages, one in Spanish and one in Russian, so it is challenging to consider the comparison representative of each of these languages.

Nonetheless, we can highlight what we see in those single examples:

In the call held in Russian, the topics brought up were general Wikimedia topics (file formats on Commons; Abstract Wikipedia; events outreach; etc.) similar to topics in English-language calls, including questions to which answers are readily available in written documentation (and even actively communicated as announcements) in English; this perhaps suggests a language barrier: not necessarily in the sense of inability to understand the material in English, but possibly in that there is no active listening or seeking of the material in English.

In the call held in Spanish, the conversation focused much more strongly on specific needs of the Spanish communities and their relationship with the Foundation, using the call to convey specific requests from the Foundation, rather than asking general Wikimedia questions. A major concern that was communicated at that call is that Spanish-community attendees would like to see it be the role of someone at the Foundation, specifically, to think about and support proactive community development of the Spanish (and other) communities, and that it was unclear to them whether such clear responsibility is assigned at the moment to anyone, and if so, to whom.

Again, this cannot be taken to represent what the entire Spanish or Russian communities care about, and can easily be an artifact of how those specific conversations happened to start.

Impact beyond live calls

According to the Massviews tool, a Wikimedia Clinic digest is read an average of 30 times per day. The total page views of digests so far is, as of this writing, 4166.

Evaluation

Quantitative evaluation of this program is difficult, because one of its design goals has been to avoid burdening volunteers with surveys or registration.

Nonetheless, this being a pilot project created as an ad-hoc response to the COVID-19 pandemic, we try to evaluate it versus its design goals, and to assess whether the impact we are able to estimate is worth the investment. Based on these assessments, we make recommendations in the Recommendations section.

Versus design goals

• Easy participation: no registration was required to participate, enabling spontaneous and last-minute participation. Reminders were sent on the day of each call. Other than the one call where we tried using the free-software Jitsi-based Wikimedia Meet, which failed severely to handle the ~70 people we had on the call, no one reported difficulty participating in the calls.
• Being of service: people who attended the call were all appreciative of the calls, and found them useful. We did not notice people disconnecting before the calls were over, which also suggests people were finding them a good use of their time. However, the need we hypothesized for answers to technical or programmatic questions, or to present work for feedback, seems a lot smaller than we had anticipated. Most attendees did not bring issues of their own to the calls, and there were frequent lulls where no one had anything in particular to bring up after a particular topic was concluded. It seems most attendees came to listen, not to ask or share.
• Accommodating volunteers: the scheduling of calls at different days of the week, different time slots, and even on weekends, was appreciated by volunteers.
• Creating lasting value: the readership of the digests, far exceeding the number of people who attended the call itself, is a reassuring signal of the usefulness of having processed notes, edited for clarity and conciseness, and augmented with links and follow-ups. Without surveys, we don't have a good way to assess the actual impact of the digests on volunteers, beyond the page view count.
• Safety: While we can't be certain no complaints have been filed with Trust and Safety, since those are highly confidential, we have not observed any violations of the Friendly Space Policy during any of the calls. Notably, discussion was calm and polite even when topics in controversy elsewhere were raised (e.g. the Brand project, the Universal Code of Conduct project).

Impact versus investment

As described in the context section above, the program was conceived at a time of uncertainty both external (the pandemic) and internal (the Foundation's delayed response to planning due to the continuing impact of Covid-19), and was designed to depend on little more than staff time of the Community Development team. Hosting in other languages, however, did require us to informally recruit volunteers fluent in those languages from among staff, which was an additional, unbudgeted, investment in the program, albeit a relatively small one (approximately 4 hours total from each person, including help with translation of announcements and tidying up the raw notes).

The Community Development program officers spent time scheduling the calls, coordinating presenters for the scheduled segment, posting and tracking announcements in almost a dozen public channels, facilitating the calls themselves, taking notes during the calls, processing the raw notes into digests, doing any required follow-up based on questions or requests made during the call, capturing the follow-up into the digests, and posting the digests on Meta and announcing them on the public channels.

All told, we estimate each call required approximately 17 hours of staff time.

Without a value placed on the intangible benefits of maintaining contact with several handfuls of volunteers using a live video call in these times of pandemic, it is impossible to determine whether that investment is worth it. We would have felt more confident asserting it is worth it had we seen more than a hundred volunteers regularly make use of the opportunity, but the numbers suggest that the calls are considered useful by only a small fraction of even the several thousands of volunteers our public channels reach.

Recommendation

While the program is useful to volunteers and well-received among communities, we recommend greatly reducing the frequency of the Wikimedia Clinics program. The Community Development team is small and growing its portfolio, and tasked with impactful and ambitious projects, chief among them the Online Learning Pilot. That project's demands and challenges are still only estimated, and it would be prudent to allocate staff time and energy to ensuring it is successfully launched.