Wiki Education Foundation/Outreach Pilot final report

The Outreach to High Achieving Students Pilot began as a response to the success of Wiki Ed's Classroom Program. Since 2010, the Classroom Program has proven that student editors will produce high-quality Wikipedia content within the classroom context. But what happens when students are asked to edit outside the classroom? Through this pilot, we sought to help students create unique projects that bridge academic and personal interests while filling Wikipedia’s critical content gaps. The main goal of this pilot was to find out if university students in the U.S. could be engaged in improving Wikipedia content as an extracurricular activity.

Key questions we sought to answer

Will students contribute good content to Wikipedia as part of their extracurricular activities?
How enthusiastic will students be about editing Wikipedia outside of the classroom?
How will their productivity compare to the productivity of students in our Classroom Program?

Specifically, the pilot project was asked to:

target high achieving students
evaluate the viability of extracurricular student editing and find ways to predict future success in this area

The following activities were deemed beyond the scope of the project:

creating student clubs on campus
making editing Wikipedia "fun for fun's sake"
asking students and the community to come up with goals on their own
increasing brand awareness or creating affiliation

The purpose of this document is to summarize the results of the 6-month pilot and to provide recommendations for next steps.

Assumptions

We understand that extracurricular editing is competing for students' time with other priorities. For this reason, our target student group audience includes those who we consider "high achieving" in their field as well as those with the potential interest in Wikipedia editing.

Originally, we defined high-achieving students as students who were invited to join honor societies in their discipline due to course completion and GPA requirements. We later expanded our outreach to include student groups studying any subject who met our criteria for selection, the details of which are covered in the next section, "Preparation." We believe students involved in honor societies and other extracurricular student clubs are meeting to discuss topics they are passionate about. We posited that passion should lead to high-quality contributions to content gaps in their fields, and may lead to students continuing to edit Wikipedia after the term ends.

Hypothesis

We hypothesized that high-achieving students in the U.S. might provide high-quality contributions to Wikipedia during their extracurricular time and that they might be enthusiastic about the project.

Preparation

In November 2014, Wiki Education Foundation hired an Outreach Manager to plan, execute, and evaluate the pilot program. During the planning phase, which concluded in January 2015, the Outreach Manager got support from the senior management and the communications teams, and consulted with experts in the field, including Pete Forsyth of Wiki Strategies.

Tasks completed during the preparation phase of the pilot included:

Development of outreach materials for students
Creation of workshop presentation template and script
Research and outreach to individual student clubs as well as to academic associations and honor societies

A two step criteria for selection process was also created in order to guide the identification of potential student clubs as well as the decision making around club commitment. The first step of this process included finding clubs that met two specifications: first, were they located in California specifically or on the west coast generally, and second, were they working in disciplines that supported species, photography, and editing or proofreading assignments. Once clubs were identified that met both these specifications, we began our cold emailing and outreach. As interested student groups were identified, we then used our second criteria for selection metric. We created this tool in order to predict the success of a student club in our program. In it, we rated each student club out of a 50 point total, taking into consideration six criteria, including response time and communication, enthusiasm, mission alignment, leadership quality, geographic location, and project quality. We considered clubs that reached a point total of 41 or above to be good candidates for the term. More about this index can be found in Appendix 1.

From December 2014 to March 2015 we reached out to a total of 190 individual student clubs. These clubs were led by students, and sometimes, overseen by an advisor. Of the 190 that we contacted, 49 student chapters were affiliated with honor societies involved in the Association of College Honor Societies. The remaining 141 student chapters were subject specific but not affiliated with an honor society. Our goal this term was for at least 30% of our outreach to be honor society specific. In actuality, 25% of our student club outreach was directed to honor society chapters.

As a result of this student club outreach, we achieved a response rate of about 16% from student clubs, or 30 out of 190. This response rate includes those who replied that they were not interested, those who replied they were interested in learning more, and those who wanted to learn more but who we later decided not to work with. Of the 30 clubs that responded to our outreach, we decided to move forward and work with 6 individual chapters. This represents an overall conversion rate of 3%.

In January of 2015, we decided to expand our outreach to the actual honor society organizations. We felt that Wikipedia editing's strong compatibility with community service would be ideal for creating buy-in from the national level. Throughout the spring term, we reached out to 24 national honor society organizations and achieved a response rate of 46%. We consider this response rate to be much higher than anticipated with 11 out of 24 organizations responding.

Of those contacted, 25% or 6 out of the 24 honor societies, sent out newsletters or email blasts to their student members on our behalf. From this top down outreach, we got in touch with two individual student chapters that were interested in learning more about our program. Of the two groups we got in touch with, we moved forward and ran our program with one of them.

Learnings: Program Plan, Timeline, Outreach

Initially, we believed that conducting our outreach to student clubs in December and January could result in 16 student club commitments by the end of January. By the end of January, however, we had contacted 140 individual student clubs and had commitments with only two to move forward. Looking into the reasons for this unexpectedly low result, we learned that many student clubs had already planned their spring term commitments and did not have the capacity to take on new projects.
We chose not to work with two clubs who had a tendency for advocacy work. As a result of this initial choice, none of the students who chose to edit got in conflict with the existing community of Wikipedians.
Academic honor society organizations seemed to be very interested in our work. Almost 50% responded to our cold emails, and of those, 50% chose to help promote our program. We assume this can be attributed to honors society organizations having full and part time staff dedicated to answering emails and following up on student engagement work. However, even though the staff responded, and often promoted our program in some way to their membership, only a few students actually responded to that outreach.
When asked to identify a few of their high performing chapters on the west coast and share that information with us, all national headquarters of honor society organizations were unwilling to share student emails with our team due to privacy concerns regarding sharing student emails with outside organizations. When asked to share our contact info directly with these chapters instead, the honor societies again faltered. In follow up, we found out that most societies felt that students already had a lot of required activities on their plate. While the national chapters thought working with Wikipedia was a great opportunity, they did not want it to feel mandatory. A direct email from their student engagement coordinator might put too much pressure on their students and make them feel required to participate. They felt most comfortable sharing blog posts and newsletters because this allowed students to feel a sense of ownership with their participation and contact Wiki Ed on their own time.
The M&R Benchmark Study reports that for fundraising efforts, non-profits receive an average click through rate for email communications of 0.5%, and a response rate of 0.06%. With this point of reference, our 16% response rate and 3% conversion rate was much higher than anticipated. It is worth noting that the benchmark study is reporting on fundraising emails in this instance, which is asking for people's money. In our program we are asking for people's time, which we think is a comparable ask. Also, these programs are using interest lists for their emails, while we were doing cold emailing. Both of these points further support our claim that our response rate of 16% and conversion rate of 3% are a huge accomplishment.
One major additional learning here, however, is that the success of our outreach does not equal a success in content. The initial enthusiasm as measured in our response rates and conversion rates initially led us to believe we had a high probability of success in terms of content contributions. However, this did not play out by the end of the term.

Communication Materials

At part of the pilot, we created outreach email templates and informational flyers for the academic associations and for students. We also ordered promotional materials as a thank-you for students who participated in our activities.

Learnings: Communication

When creating the outreach email templates, we started with an A/B test for our student outreach. The first email, the A test, was a shorter email with less information about the pilot program. The second email, the B test, was a longer email that included a short history of Wiki Ed along with in depth coverage of potential student benefits of participating in the pilot program. As a result of our testing, the shorter email resulted in a much higher response rate.
Branded clothing during outreach sessions can give students the impression that they are being 'sold' a product and should be avoided.
After our first and second workshop, it seemed clear from our survey feedback that students wanted more stories to help frame the Wikipedia community for them. We worked to reframe the workshops with a new introduction, in which we shared a "Wikipedia story" with those present. We chose one that acknowledges Wikipedia's global readership, as well as the capacity for Wikipedia to change lives. This frames student Wikipedia projects in a new way, allowing them to feel connected to the community and to their work on a new level.

Execution

During the execution phase of the pilot, we tried to engage students through workshops and field trips. Prior to meeting with the students, we worked with the club leader to discuss academic and personal interests of the club members. Based on this information, we identified potential content gaps where student editors could make an impact both visually with photo uploads as well as in writing and proofreading work.

During our workshops, we tried to target these club-specific content gaps to help students see the direct impact they can make. We found topical news stories that highlight current events surrounding a related Wikipedia page and presented both sources to the student. Then we asked students to think about what information from the news source could benefit the Wikipedia coverage of this topic. Once the students found a gap, we encouraged them to be bold and make the edit. During our workshops we also asked students to think about what editing qualities they think would make a good or featured article (clarity, well written, neutral, verifiable, and comprehensive coverage of topics are a few) as well as to think about what kinds of images would best represent certain article topics. Thematically, we tried to focus on the direct impact our students can have, both for their own skill sets as they develop their writing skills, media literacy, critical thinking, and community service hours, as well as from a global perspective. We want our students to know that their work can and will make a difference to one of the world's most visited websites.

Our field trips hoped to explore whether these types of activities would encourage students to upload images to Wikipedia and also increase the students' overall enthusiasm about adding content.

Between February and May 2015 we worked with the following student clubs:

University of California, Berkeley Berkeley Water Group Idea Lab (2 workshop sessions, 1 one-day field trip planned and executed): 12 attendees total
University of Arizona GeoClub (1 workshop session, 1 half-day field trip planned and executed): 13 attendees total
University of California, Santa Barbara Art, Design, & Architecture Museum Club (1 workshop session, 1 half-day field trip planned, but not executed): 11 attendees total
Portland State Lambda Pi Eta (1 workshop session, no field trip): 2 attendees total
Oregon State Hydrophiles (1 workshop session, no field trip): 4 attendees total
Oregon State Pi Alpha Xi Horticulture club (1 workshop session, no field trip): 4 attendees total

Learnings: Execution

At our first workshop we did not bring a sign in sheet, or have links to our course page and online training for students to access during the workshop. In later workshops, we used a web shortener to create shortcuts for students to enroll in the course page, we brought a sign in sheet, and we later began bringing photo release forms for students to sign during workshops in order to avoid follow up. We also instituted a post-workshop survey in order to continue making iterative improvements to our workshops.
We had dinner with one of the student groups in order to find out whether social bonding would lead to more student engagement once we left the site. In reality, only a few students made edits after we left, and only one student made any significant edits at all.
As part of our A/B testing we found out that field trips lead to a higher number of attendees, but not to an increased outcome when it comes to the students' contributions to Wikipedia.
Engagement with student groups can benefit from partnerships with local institutions that have an interest in quality information about their specific field on Wikipedia. East Bay Municipal Water District helped the Berkeley Water Group Idea Lab by providing learning materials and staff time for their field trip. Peavy Arboretum staff in Oregon attended the student learning workshop, brought materials, and are now working with the group to help support collaboration between the two organizations.

Learnings: Online Training

Of 44 students, 16 completed the online training (36%).
When working with students, getting them to complete the online training continues to be one of Wiki Ed's biggest hurdles. The Classroom Program this spring saw a completion percentage of 52%, a significant increase when compared to previous terms (spring 2014: 25%, fall 2014: 28%). The outreach pilot saw a completion percentage of 36% in its first term. We would like to increase these numbers for all our student programs in 2015–16.
We believe the completion percentage in the outreach pilot occurred for a couple of reasons. We had some student leaders express concern that the list of requirements for students to complete before the date of the workshop was too long. This list included creating a username, joining the course page, and completing the online training. The student leaders expressed that this felt like homework and showed concern that students would not complete the tasks, especially the online training which we estimate takes students 30 minutes to complete. We adjusted our workshops to include a time for students to create their usernames and enroll in our course pages, but we did not include a time to complete the online training. If we make the online training a mandatory pre-requisite for student editing and participation in this program, we may find that we have no student groups to work with.

Content Goals & Outcomes

Goals & Contributions

Our goal this term was to work with 16 student clubs. Assuming that a club would consist of 10 students on average, we set a target of having 4 out of those 10 students edit Wikipedia. We set a content target of 3 kB of text added to Wikipedia and 10 images uploaded to Commons per active editor.

	# students total	# active editors total	# characters added per club	# images added per club
	Participation		Content
Target	160	64	12 kB (36,864 characters)	40
Actual	44	24	3.3 kB (3,336 characters)	21

We did not hit either our participation or content targets.

Total student edits this term from club related work

44 students
60,083 characters added (~20 kB of content)
46 articles improved and 3 articles created
21 images uploaded, 10 of which are in use on Wikipedia

Learnings: Content Goals

While we did have 46 articles edited (about one article per student), the overall amount of content contributed to Wikipedia was low. However, we were very impressed with the quality of edits we received from specific groups. The two clubs with whom we worked most closely to run a field trip contributed the most content. In each case, we planned and attended the field trip in order to encourage student editing and create enthusiasm. It seems that involving a field trip or at least an opportunity for active participation will not only increase the number of attendees but also the students' engagement level when it comes to contributing high-quality content.

Survey results

We used two surveys this term to track progress with our students and workshops. The first was a post workshop survey handed out to 4 student clubs immediately following the end of a workshop. This survey was meant to gauge what went well and what didn't during each workshop, and to ask the students to help us improve. We used the feedback from this survey to make iterative adjustments on out workshop model, including adding more Wikipedia stories into our workshops as well as a photo uploads tutorial.

The second survey was sent out electronically to all students at the end of the term. This survey was meant to learn from students what their experiences on Wikipedia had looked like and what their editing goals were for the future.

The details from these surveys can be found in Appendix 2.

Learnings: Survey data

In our end of term survey, 10 out of 17 students reported that they had edited since the workshop. In the previous section, we reported that our numbers indicate only 5 students edited after their respective workshops. This discrepancy could result from students forgetting their passwords and creating new user accounts that we are not tracking in our dashboard or from students making edits when they are not logged in.
The survey response rate was 38%, or 17/44 of our student editors from this term. The information gleaned from their responses, while valuable, is not statistically relevant due to the low number of respondents. Even with a 100% response rate, the data gathered from this pilot would not be considered statistically relevant.
The workshop survey indicated that the biggest hurdle for student editors was time. Seven out of 17 students reported that they had not done any editing since the date of their workshop. Of those, 71% or 5/7, cited being too busy as their primary reason for not editing. When asked what might prevent them from editing Wikipedia in the future, 81%, or 13 of 16 respondents said they are worried that they won't have time to contribute. These responses indicate that Wikipedia editing is low on students' priority lists. If we want to increase the number of active editors, we need to find a way to move Wikipedia editing higher on these priority lists. We recommend that when there is available capacity, Wiki Ed does a focus group or new pilot to determine how to move Wikipedia up on student priority lists.

Conclusion

This 6-month long pilot answered several questions. First, whether students at U.S. universities would edit Wikipedia as an extracurricular activity. Second, whether they would improve Wikipedia's quality. Third, how enthusiastic would students be about editing Wikipedia in their free time?

It's clear that U.S. students can improve Wikipedia's quality in a classroom setting and in their free time. However, our pilot suggests that, without additional incentives, most of these students choose not to do so.

The greatest challenge in targeting extracurricular editing was the absence of external incentives. Our A/B tests show that field trip participants were more motivated student editors. Without field trips as an incentive to spark that motivation, however, participation in Wikipedia-related activities dropped. With a grade, field trip, or staff visit, students contribute to Wikipedia. Without those incentives, they contribute significantly less, or not at all.

Unfortunately, field trips are not a scalable way of incentivizing students. We see greater opportunities in models that rely on alternative, scaleable solutions. For example, honor societies or academic associations could incentivize Wikipedia editing by counting editing time as a community service.

From the Wiki Education Foundation's perspective, this pilot was a success. We answered the questions we set out to address. While our pilot generated less data than we had expected, we are confident that further experimentation with more participants would confirm our findings. For this reason, and with the amount of learnings we collected, we will not be extending the pilot beyond spring 2015.

Appendix

Appendix 1

Criteria for Selection – part 1

Student clubs and honor societies had not been previously identified, so the bulk of the work during November, December, and January included researching and contacting those who met our selection criteria:

Geographic Location: We chose to focus on student programs located on the west coast in regions such as California, Arizona, Washington, and Oregon. We made this decision because these locations cut down on staff travel time from our office in San Francisco, CA, and because the warm climates allowed for species editing and photography projects.
Types of student groups: We chose to keep our focus on student groups studying academic disciplines that supported species, photography, or editing and proofreading projects. We chose these because we believe editing in species/classification spaces is made accessible through the use of templates, uploading images to Wikipedia and adding them to article spaces is an easy skill to learn and holds the potential to have huge impact, and because we already have the capacity through our Classroom Program to support writing assignments.

Criteria for Selection – part 2

After we received interest from a student club and discussed the goals of the project, it was time for us to make a decision about which clubs to move forward with. In order to make this determination, we came up with an enhanced "Criteria for Selection" that would help us project the success of a student project. This projection includes an evaluation of the following criteria for each individual club. The criteria are rated out of a total score of 50. Some criteria are rated out of 5, while those criteria rated out of 10 were seen as "high priority" criteria.

Response time & Communication ( R ) : Rated on a scale of 1–5, this is a representation of the communication skills and response time of the student leader. A score of 5 represents a leader who was quick and efficient in their communication, and a score of 1 represents a leader who was slow to respond and who exhibited poor writing or communication skills.
Enthusiasm ( E ) : Rated on a scale of 1–10, this is a representation of the engagement and enthusiasm of the student leader. A score of 10 represents a leader who was highly engaged, asked thoughtful questions, and showed excitement about the opportunity. A score of 1 represented a leader who exhibited no visible excitement and little to no engagement during Skype or phone call discussions.
Mission alignment ( M ) : Rated on a scale of 1–10, this is a representation of the students' excitement about Wikipedia. A score of 10 represents a student leader who seems to be a champion of Wikipedia and is excited about closing content gaps in their field. A score of 1 represents a student who is questioning of Wikipedia and its role and who does not see the benefit of student contributions for the student editors or for the community.
Leadership quality ( L ) : Rated on a scale of 1–10, this is a representation of the overall organizational goals of the student leader and the student club. A score of 10 represents a leader who is very engaged, has a goal for the club for the term that Wikipedia can help them reach, and who is a good spokesperson for themselves and their club. A score of 1 represents a leader with little to no understanding of the goals of the club, who is not engaged with their membership, and who does not act as a spokesperson for themselves or their club.
Geographical Location (G): Rated on a scale of 1–5, this is a representation of the ease of travel to the club location. A 5 is a club that exists within 2 hours driving distance, while a 1 is a club that requires a cross country flight.
Project Quality (Q): Rated on a scale of 1–10, this is a judgement on the potential quality of the editing project decided upon by the student group and the outreach manager. A score of 10 represents a project that meets a content gap goal, is reproducible in type and scale for future terms, and seems achievable. A score of 1 represents a group that is unable to name a project theme.

Each club was rated on all these criteria in order to project the success of their student project. The table below is an example of how we stored this information. In the execution section, you can see how we scored the clubs we moved forward with. We believe that clubs scoring 44 and above can and should contribute high quality content.

Club Name	Geographic Location (1–5)	Leadership (1–10)	Mission Alignment (1–10)	Response Time/ Communication (1–5)	Enthusiasm (1–10)	Project Quality (1–10)	Total Score (Out of 50)	Project Details & Notes
club name	city - location score	leadership score	mission alignment score	cmn score	enthusiasm score	project score	total score	notes

Appendix 2

Post workshop survey

Overall, we had 20 students fill out this survey. From it, we learned that the most helpful part of the workshop was the editing exercise (17). Other responses detailed the most helpful part of the session to be the question and answer session (1), help on creating an account and a demonstration of a photo upload (1), and getting to see the Wikipedia facts around gender gap and content gap work (1). The least helpful part of the session was identified to be the background on Wikipedia by 2 of the students, while the other 18 felt that the entire session was helpful. When asked what students wished we had covered, we found that from the first workshop, students wanted more funny stories and experiences about Wikipedia, something we took into consideration and later adapted into our workshop communication. Other things students wanted to hear included how to create a page (1), getting to upload a photo themselves (1), more about the history of Wiki Ed (1), more instruction on how to find articles to edit (1), and more time to learn about formatting and practice editing (2). We began doing uploading tutorials at our later workshops with this feedback.

On a scale of 1-5, 1 being very easy and 5 being very hard, students were asked how easy to understand the session was.

--	Score of 5 – very hard to understand	4 – hard to understand	3 – neutral, not easy or hard to understand	2 – easy to understand	Score of 1 – very easy to understand
Number of votes	5	2	0	4	9

One student commented that the language of Wikipedia was new to her, which made it difficult to follow along. But overall, the majority (65% or 13/20) said the workshop was easy or very easy to understand.

End of term survey

We had 17 students respond to our end-of term- survey as of June 4, 2015. This is a 38% response rate.

Students were asked if they had done any editing following the workshop. 58.8% of students responded that they had edited since the workshop, (10/17 YES, 7/17 NO). The top reason for their continued editing was cited as "sharing knowledge is important to me" at 50% (5/10). Other responses included "I enjoy sharing information on a topic I'm passionate about" (20%, 2/10); "I feel I have a lot to contribute" (10%, 1/10); and "it's fun" (20%, 2/10).
Those who had not continued editing (41.2% of students) cited being too busy (71.4%, 5/7) and editing being too difficult (14.3%, 1/7). All our questions also included the option to write in a response. One student replied "My expertise would be where I choose to edit/add content and it might be self promoting (14.3%, 1/7) as a reason for not contributing. Other response options included "I don't have anything to contribute," and "I don't see the point in editing Wikipedia." Neither of these options received any responses.
When asked if they had uploaded any photos to Commons, 17.6% of responses said they had and 82.3% of responses said they had not added photos. Those who did upload photos reported that they did because they thought it illustrated a topic well (66%) and because they liked having a copy of their photo online to share with friends (33%).
Many students said that they had added photos either at the workshop or after. When asked if they added their photos to a Wikipedia page, 75% of respondents who said they uploaded a photo said yes. 100% of those students reported that they did this because they liked making Wikipedia a better resource.
Those who had not uploaded any photos reported that they hadn't because they were too busy (42.8%, 6/14), they didn't have photos to contribute (42.8%, 6/14), or that they intended to but were too busy (14.3%, 2/14). This last response was a write in response from two students.
When asked about their editing potential in the future, 100% of respondents or 17/17 students reported that they would edit an existing article in the future; 58.8%, or 10/17, of all respondents said they would create a new article in the future; 82.3%, or 14/17, respondents said they would add a photo in the future; 58.8%, or 10/17, said they would add references to an existing article in the future; and 35.3%, or 6/17, said they would copyedit an existing article in the future.
When asked what might prevent them from editing Wikipedia in the future, 81.25%, or 13 of 16 respondents said they don't have time to contribute; 12.5% or 2 of 16 said they can't think of anything to contribute; and 6.25% or 1 respondent said that editing Wikipedia is not fun for them.

End of term survey results

(as of June 4, 3:20pm) - 17 total responses (out of 44 potential)

Have you done any Wikipedia editing since the workshop? 10/17 YES (58.8%), 7/17 NO (41.2%)
If yes, why did you continue to edit Wikipedia?
- 5/10 (50%) Sharing knowledge is important to me.
- 2/10 (20%) I enjoy sharing information on a topic I'm passionate about.
- 1/10 (10%) I feel I have a lot to contribute.
- 2/10 (20%) It's fun.
If no, what factors prevented you from editing Wikipedia?
- 5/7 (71.4%) I'm too busy.
- 1/7 (14.3%) It's too difficult.
- 1/7 (14.3%) My expertise would be where I choose to edit/add content and it might be self promoting.
Have you uploaded any photos to Wikimedia Commons since the workshop? 3/17 YES (17.6%), 14/17 NO (82.3%)
If yes, Can you tell us why you uploaded photos?
- 2/3 (66%) I thought it illustrated a topic well
- 1/3 (33%) I liked having a copy of the photo online to share with friends
If you uploaded a photo, did you add your photo to a Wikipedia article? 3/4 YES (75%), 1/4 NO (25%)
What motivated you to add your photo(s) to Wikipedia?
- 3/3 (100%) I like making Wikipedia a better resource
If you haven't uploaded a photo since the workshop, can you tell us why?
- 6/14 (42.8%) I'm too busy
- 6/14 (42.8%) I don't have photos to contribute
- 2/14 (14.3%) I intend to but keep putting it off because I have been busy.
Do you think you might do any of the following activities in the future?
- 10/17 (58.8%) Create a new article
- 14/17 (82.3%) Add a photo
- 17/17 (100%) Edit an existing article
- 10/17 (58.8%) Add references to an existing article
- 6/17 (35.3%) Copyedit an existing article
What might prevent you from editing Wikipedia in the future?
- 13/16 (81.25%) I don't have the time to contribute
- 2/16 (12.5%) I can't think of anything to contribute
- 1/16 (6.25%) It's not fun for me