Wiki Education Foundation/Summer Seminar Pilot final report

Background

While Wikipedia is one of the most widely consulted general references, its subject matter coverage is uneven and idiosyncratic. Topics that attract most interest from readers are often poorly covered As one recent research paper shows (Archived version), psychology is a field which receives a significant amount of page views but which is underdeveloped in terms of quality content. Medical topics (including psychology) are particularly challenging to edit on Wikipedia due to advanced sourcing requirements, and this can often deter new editors, even experts, from contributing.

But psychology instructors are interested in improving Wikipedia's content in their topic area, as can be seen from the enthusiasm for Wiki Ed's partnership with the Association for Psychological Science (APS). While we've always encouraged faculty experts to contribute content to Wikipedia along with their students as part of our Classroom Program, we haven't had much success in getting instructors to write articles; instead, at most instructors will collaborate with their students on articles the students are improving for class. Nevertheless, APS, Wiki Ed, and many Wikipedia editors would like to see more subject matter experts write content for Wikipedia.

While editor retention is a challenge for most programs on the English Wikipedia, the School of Open course ("WIKISOO") run by Pete Forsyth indicated that a model of encouraging self-selecting newbies to add content to Wikipedia through a structured online class environment has potential. Pete also met with us early in the planning stages to share his learnings from running the WIKISOO course four times; we thank Pete for his willingness to share his insights.

With the encouragement of APS, we launched a pilot Summer Seminar in Psychology to evaluate the potential for instructors — as experts in their field of study — to contribute content to Wikipedia through a structured one-month "class" run by the Wiki Education Foundation.

Theory of Change and Research Question

Instructional faculty in higher education are a group of subject-matter experts with an ongoing interest in their field of study. They also have a depth of understanding of the technical literature in their field, and a commitment to the dissemination of accurate information about their field of study to the wider public that make them ideal potential contributors to Wikipedia. While many of the instructors that Wiki Ed works with have expressed interest in contributing to Wikipedia, a lack of familiarity with Wikipedia's policies and processes, and with the Wikipedia community, can be a hurdle for instructors who are interested in editing. Given a structured setting in which we instructed faculty in how to edit Wikipedia, we sought to determine if these instructors would contribute substantial content to Wikipedia.

Additional questions we hoped to answer included:

What are best practices for encouraging subject experts to edit Wikipedia?
What are the hurdles that prevent experts from editing?
Which types of subject experts express interest in editing and which actually make edits?
Can a structured course bridge the gap between enthusiasm and actual editing?
Do instructors who contribute regularly to Wikipedia run more successful Wikipedia assignments?
What other programs might encourage experts to contribute?

Program Logic Model

We find it helpful to break down this evaluation of the pilot program into a Program Logic Model. By using this lens, we can determine whether our pilot has the potential to have impact, in alignment with the theory of change we outlined above.

Inputs

The major inputs for the Summer Seminar were the brochures mailed to program participants, the Dashboard program we used to track program participants, and the staff time that went into planning and executing the course. The brochures and Dashboard were existing resources; the major expenditure was staff time. Approximately 60 hours of staff total went into the program in total.

Activities

The Summer Seminar was modeled as a condensed version of the standard Wikipedia assignment seen in our Classroom Program. The syllabus for the course was generated by Wiki Ed's assignment design tool hosted on dashboard.wikiedu.org. The timeline for the seminar was hosted on the Seminar's Dashboard course page.

The Seminar was planned to run from July 27 to August 28, 2015. Participants were able to enroll until July 24. Students who enrolled before July 15 were sent printed copies of Wiki Ed's Editing Wikipedia and Editing Wikipedia Articles on Psychology brochures. All handouts and course materials were available digitally on the course page. The course was co-run by Helaine Blumenthal, Classroom Program Manager, and Ian Ramjohn, Wikipedia Content Expert, Sciences.

Outputs

Twenty psychology instructors registered for the class. Eight were senior faculty (professors or associate professors), five were junior faculty (assistant professors) and three were graduate students. Fourteen of the participants were female, six male. Three of them had taught previously with Wiki Ed, and six of them had edited Wikipedia before signing up for the seminar.

Class structure

We communicated about the course via email with all participants. The Seminar was created as a largely self-directed course. Participants were expected to complete a series of assignments each week of the program. The assignments for the first week were mostly preliminary training exercises, including Wiki Ed's training for students.

During the following four weeks, we scheduled a weekly Google Hangout in which participants could interact directly with Ian and Helaine. The students were encouraged to submit questions to Ian and Helaine prior to the hangout. The weekly meetings were initially intended as discussion forums in which participants could ask questions that they encountered as they completed different milestones in the course. Due to lack of participation in the first hangout session, Ian and Helaine decided to provide a brief lecture on relevant topics for the remaining Google sessions and leave a smaller portion of the time for Q&A. We cancelled the final planned session due to lack of participation.

At the end of the course, we distributed an end survey to all participants. The goal of the survey was to ascertain their views of the seminar and to compare their responses to the intro survey. Only four of the participants in the course filled out the post-course survey. Three of the four indicated that they completed the editing assignment, while three participated in at least one of the Hangouts. Although the sample size was small, all four participants thought that the seminar was a worthwhile use of their time.

Pre survey outputs

We asked participants to fill out an introductory survey to gage their Wikipedia editing experience prior to the seminar and to better understand their expectations for the course. Seventeen of the 20 participants completed the pre-survey.

Editing experience

Sixty-five percent of participants reported that they had no prior experience editing Wikipedia; only two participants (12%) had moderate experience and none of the participants had significant editing experience.

Motivation

Based on the pre-course survey, participants in the seminar were primarily motivated by a desire to learn more about editing Wikipedia and to learn more about using Wikipedia as a teaching tool (8 of 14 respondents, in each case). Other motivations expressed included a desire to learn more about Wikipedia policies, community norms and the culture of Wikipedia (five respondents) and to learn about more about article quality evaluation (two respondents). Two participants also expressed an interest in getting to know others with a similar interest. (See Appendix I for detailed responses.)

Wikipedia edits

Fourteen participants completed the online training for students. As part of that training, participants are guided through making edits in their sandboxes. These edits are not included in the statistics below.

Sandbox content additions

Eight program participants added content to their sandboxes totaling approximate 30,000 bytes (about 20 printed pages of content). Most userspace contributions were drafts for incorporation into articles, although one was an 826-word, 14-reference draft of a new article which is nearly complete enough to move into mainspace, although it had not been moved as of this report writing.

Mainspace content additions

Participants contributed a total of 8,588 bytes to English Wikipedia articles (the equivalent of about 5 3/4 printed pages of content). Six participants edited in the article mainspace; only one of these editors had never edited before. The largest contribution was a 266-word and eight-reference article expansion. Other class contributions included the creation of two new articles (64 words and 71 words respectively).

Outcomes

While participants made very few contributions to Wikipedia, the quality of their contributions was high. Contributions were well-written and well-referenced. References were well formatted. Participants used high-quality sources, and used current literature (except when discussing historical events). This suggests our course content was effective at teaching participants how to contribute effectively.

Conclusion

Though participants were enthusiastic and eager to learn about Wikipedia, their contributions were small during the Seminar. The quality was high, suggesting that involving experts can lead to quality content; however, we believe the staff time inputs do not justify the output and outcome of this pilot program. Therefore, we are unlikely to pursue a similar program in the future.

We have some hypotheses about the lack of contributions:

Nearly half of pre-survey respondents indicated they wanted to learn more about editing Wikipedia. The vast majority of participants completed the online training — which provides a good overview of Wikipedia policies and the basics of editing, designed to give student editors in our program the knowledge they need to know about Wikipedia to effectively contribute — but never followed through with another activity. The online training may have provided the information participants were seeking, and they didn't actually want to contribute themselves. We sought to answer this with the post-survey, but we did not have enough responses to draw any conclusions.
Another half of respondents indicated they were interested in teaching with Wikipedia. They may have been confused about the Seminar's purpose (to encourage them to edit themselves), and when investigating further, determined that the course wasn't what they were expecting and stopped participating. This suggests we should revamp our instructor training to make it more accessible, since these instructors seemed to want to learn how to teach with Wikipedia.
The mode of delivery was problematic. Participants had difficulty understanding how to sign into the discussions. This was further complicated by the fact that there were two classes of listeners — participants, who could speak, and listeners who could only interact via the Q & A tool. Only one of the three Hangouts had good interaction with the listeners. Complications with the delivery method may have soured participants on the activity overall. With better structure, the delivery of the discussions could be greatly enhanced. We would also do well to expect more of the participants in the seminars — set several milestones that they should meet — and do more to foster communication between participants (e.g., assigning pairs to work together or peer grade each other's work). While worth noting, we don't believe these technical difficulties had significant impact on activity; at most only seven participants tried to join the simultaneous sessions, even though 11 had indicated those times worked for them during a pre-class survey for good times.
While APS encouraged their members to participate in the Seminar, having an explicit reward at the end (some sort of officially recognized designation or such) from APS could perhaps have provided enough external motivation to keep participants engaged.

Enthusiasm for learning more about Wikipedia and a desire to help Wikipedia were definitely present, and can be seen from the level of completion of the online training. Thus, we don't think it's appropriate to abandon all plans for engaging subject matter experts to edit Wikipedia. Instead, we believe future work should look to alternative engagement strategies beyond asking experts to write content. For example, we believe the enthusiasm showed by participants toward Wikipedia could be channeled more productively into shorter tasks requiring less advanced knowledge of wikicode and Wikipedia policies. In such an "Ask an Expert" program, instructors could register themselves in a database with their names, affiliations, topics of expertise, and contact information (similar to a university experts guide for media). These experts would then be on stand-by when a Wikipedia editor needs a question answered from a subject matter expert. We are interested in exploring this idea in future pilots.

Appendices

Appendix I: Motivation

Participants in the seminar were motivated by a desire to better understand Wikipedia; some were motivated by a desire to contribute to articles, while others were interested because they were planning to teach a class that involved Wikipedia editing.

Briefly describe why you signed up for the seminar.
I signed up for this seminar because I will be starting to teach in the fall semester and would like to know how Wikipedia can be integrated as a educational resource into the classroom, and how lessons can be designed around the use of contributing to Wikipedia. I am also interested in learning how to become a contributor myself, and how articles about psychology are created, edited and reviewed for quality assurance.
I wanted to learn more about editing Wikipedia entries. Both in terms of the nitty gritty of coding as well as gaining a better understanding of how to edit and create high quality articles. I want to eventually be able to share this knowledge with psychology students so they can edit/create Wikipedia entries on relevant topics as part of their education.
Learn how to edit articles, learn any relevant policies and procedures
I would like to link up with a community of scientists who are actively editing Wikipedia together! It will be interesting to relearn some things and learn from the others who have edited in this area.
- learn technical basics of editing - learn how to structure a student assignment - explore how organizational behavior content is organized on Wikipedia - learn who the key editors are for content on groups & teams - identify areas where editorial work is needed - learn the norms of conduct for the community
I wish to develop some expertise with editing articles, as well as a better understanding of the "rules of engagement" on Wikipedia (i.e., its guidelines and how to work cooperatively with others). I also wish to determine if this is something I could use with students in my honors intro-psych course.
I want to be able to effectively manage a Wikipedia editing assignment in my spring seminar course.
I apologize for having to unenroll at this time. I was really looking forward to it, but I am about to have knee surgery and start the school year simultaneously. My time is suddenly not my own. I hope this seminar will be offered again as I would really like to participate sometime. The Training for Students section alone was totally worth it!!!
I have tutored students through the process of editing Wikipedia but done relatively little editing myself. This provides me with a chance to get a better sense of how articles are evaluated by doing some myself.
To get a basic experience with editing and learn more about Wikipedia culture. Also to interact with psych instructors who are interested in Wikipedia and find ways for APS to support efforts to improve psych content.
To understand Wikipedia editing better.
Interest in potentially editing or writing articles on psychology topics or integrating into a course I teach.
I wanted to see if this could be an assignment that I could do with undergraduate students at my University.
Building the basis for using wikipedia in class next term.
I would like to teach using Wikipedia. I need to learn all that I can about this tool, so I understand what I ask my students to do. Taking this course will, I expect, allow me to experience the student role and see their perspectives better. I am comfortable with computers and online learning systems, but O have little experience with many of the newer tools like Google hangouts, etc. so I hope to learn a lot there too.
I wanted to learn how to engage with the Wikipedia community, especially in regards to the field of psychology.

Appendix II

Google Hangouts
Google Hangouts

Week 1 https://www.youtube.com/watch?v=V7okiYCSOXk
Week 2 https://www.youtube.com/watch?v=nmzKFTdLNLc
Week 3 https://www.youtube.com/watch?v=IgjruAvuk-E

Course dashboard