Wiki Education Foundation/Year of Science evaluation

The Wikipedia Year of Science 2016 was the biggest large-scale content initiative ever undertaken in the Wikimedia movement. Led by the Wiki Education Foundation (Wiki Ed), the Year of Science was a program designed to improve the availability of information in the sciences on Wikipedia. In this report, Wiki Ed staff have documented what we did as part of the Year of Science, what worked, what didn't, and what we learned. Several other Wikimedia entities also engaged in activities as part of the Year of Science campaign, but the focus of this evaluation is specifically Wiki Ed's work.

History

As part of the strategic planning process in 2014-15, Wiki Ed's board tasked senior leadership with coming up with an “audacious goal” to improve the quality and quantity of Wikipedia content. While other movement entities are focused on recruiting new long-time contributors, Wiki Ed focuses on content. In 2016, we focused on science with two major programs: Our Classroom Program, where college and university students in the United States and Canada contribute content to Wikipedia as a class assignment, and our Visiting Scholars Program, where existing Wikipedia editors get access to university sources. While Wiki Ed was the primary driver of the Year of Science, we encouraged interested community members to pursue aligned projects as well.

Why Wikipedia, and why science?

Here's how we pitched the Year of Science to potential funders:

Wikipedia has more links from search engines than any other website, and is often the first search result listed when searching for scientific information. Wikipedia's content reaches more than 450 million readers around the world, on topics ranging from greenhouse gas to the solar system. The online encyclopedia is one of the most powerful platforms for the dissemination of science information in the world.
Despite Wikipedia's reach, the interest-driven content production paradigm of Wikipedia has led to a strong misalignment between the supply and demand of quality content. While Wikipedia offers an excess of high-quality content about certain topics, a large proportion of pageviews goes to articles that are insufficiently developed. As of today, Wikipedia contains more high-quality articles about warfare than about any topic area within the science disciplines.
The Wikipedia Year of Science is designed to improve the quality and quantity of articles on scientific topics on Wikipedia.

For the purposes of this campaign, we used the National Science Foundation's definition of science, which includes so-called “hard sciences”, STEM, and social sciences. This also maps to the English Wikipedia's Science portal category definitions.

Goals

We set out the following content goals for the Year of Science as a whole:

Improved content for 5,000 science articles
Expand science coverage by 4.5 million words
Provide visual representation of science topics via 1,000 science-related images
Develop science communication skills among 4,500 college and university students
Increase coverage of the lives and works of 100 women scientists

Classroom Program

Outreach: Recruiting new classes

Early outreach activities

In order to achieve these content goals, we needed to dramatically scale up the number of classes in the science disciplines that we were working with. Prior to the Year of Science initiative, we pushed a long-term recruitment strategy: We pursued partnerships with academic associations, and part of the memorandum of understanding we signed with each association was that the association would provide us free or discounted booth space at their exhibit hall and a workshop or presentation slot at their annual conference. This meant we spent a lot of staff time up front developing these partnership agreements.

We began planning for the Year of Science in mid-2015, and Educational Partnerships Manager Jami Mathewson started reaching out to associations in the sciences. By early 2016, we'd learned that it was taking too long to develop these relationships, and we abandoned the model of formal partnership agreements and began paying for booth space in the exhibit halls of science conferences. See Appendix 1 for a list of all the conferences we attended. This dramatically increased our travel and conferences budget, but it enabled us to get enough new classes on board.

When we started out, we weren't even sure if we could bring on that many new courses. In the fall 2015 term, we brought on 89 new courses — an increase of 28 from the previous term — and we felt like somewhere around 90 new courses might be the maximum we could bring on in a term. As you can see from Figure 1, we definitely learned that we could bring on more than 90 new courses! In fact, in the Spring 2017, we onboarded 195 courses taught by first-time instructors in our program.

The cultivation cycle

The ambitious goals for the Year of Science meant that we needed to dramatically step up our recruitment of classes. To bring on new instructors, we turned to the cultivation cycle used in sales and fundraising. With the help of our Salesforce database, when each instructor gave us their email address (a “lead” in the Salesforce terminology), we put them into a specific campaign that indicated where we'd come into contact with them. We created a series of standard template emails and established a cultivation cycle to move people from first interacting with us to creating a course page and joining our program. Outreach Manager Samantha Weald managed all communication with new instructors.

As part of the Year of Science, we ended up talking to nearly 3,500 individual instructors (or leads). Most of these were people we met at various conference booths. Around 600 of these have converted into “contacts” in Salesforce parlance, which means we were engaging in individual emails with these instructors. That's a 17% conversion rate, which we think is good. We haven't previously tracked this number, so we will continue to monitor it in future campaigns.

In these communications, we used mass emails and followed by a targeted personal approach that tailored messaging to what we knew about the particular person. Systematically approaching the communication with leads and cultivating numerous points of contact has enabled us to increase the number of new courses, and doing so with the power of Salesforce enabled us to do so with the same number of staff attached to the project.

It takes time

One of the biggest learnings from outreach during the Year of Science was that the cultivation cycle takes a lot of time. As you can see from Figure 2, the impact of the Year of Science is something we only really started to see in Fall 2016, and are really seeing in Spring 2017. To do a large-scale campaign like this in the future, we recommend starting the recruitment cycles more than a year out, to maximize the time available in the cultivation cycle. Figure 2 also illustrates the challenges in our partnership-based model from early 2015. Spending our time attempting to establish partnerships with academic associations simply didn't bring on new courses in the numbers we needed to meet the Year of Science goals in Spring 2016, but after we started paying for conference booths, we were able to dramatically increase the numbers.

Visibility's impact on recruitment

A major push for the Year of Science was to increase Wiki Ed's visibility, and the visibility of our program. For several terms, we've been getting somewhere between 30 and 40 classes that come on through either a referral from a contact (someone already connected to us) or who come to us from some non-targeted outreach (they read something about us in the news media and were inspired to reach out, they had the idea to teach with Wikipedia and Googled us, a Wikipedia editor found an unregistered class and sent them our way, etc.). During the Year of Science, we wanted to increase these numbers. As you can see from Figure 3, we were able to really harness the power of visibility to increase the number of new classes. We expect this trend to keep going; one of the outcomes of the Year of Science visibility work is that more instructors than ever are aware of the possibility to teach with Wikipedia, and we expect to continue to see dramatic term-over-term growth in the number of new instructors coming to teach with Wikipedia.

Sciences vs. humanities

While we don't have data to back this up, we believe it takes slightly more effort to recruit science classes than it does to recruit humanities classes. Many science associations are recognizing the importance of writing assignments, and are encouraging their members to integrate writing assignments into their pedagogy. But writing assignments are still uncommon in classes like mathematics or physics, where students are used to turning in problem sets. We also heard pushback from some science instructors that the quality of Wikipedia in their disciplines was pretty good; once we engaged in a discussion, however, of whether it was good as a refresher for someone with a PhD in the topic area or whether it was good for a high school sophomore looking for an introduction to the topic, instructors were able to see the need for improvement. From our experience, humanities instructors “get” what we're trying to do with a Wikipedia assignment a little faster, but once we engaged with individual science instructors, we were able to showcase the importance. On a positive note, we sometimes get pushback from humanities instructors that it seems like we want to make the class about Wikipedia, not about their subject area; science instructors understood we were instead pushing for them to improve articles on science topics.

Learnings: Outreach

If we want to reap the benefits of a thematic campaign (and if we're held to outcomes within that timeframe), we need to start it much sooner (e.g., start recruitment a year out from when we want the impact to hit).
In particular, extend announcements/public relations to six months prior to when we want to have impact with courses.
Build the campaign theme into existing strategies (e.g., our current “Sustaining Science” campaign to continue the Year of Science work has gone really smoothly because we already have the leads in cultivation for science topics).
Paying for booths in exhibit halls at academic conferences is a short-term way to increase the number of leads in the database, but it is expensive: we still think that establishing partnerships is a long-term better strategy.
Following up, several times, with leads makes a huge difference.
Using Salesforce to created targeted mass messaging was a time-effective way of creating personalized messaging for large numbers of leads.
Contact referrals are a highly productive way of recruiting classes and have a low staff time investment.
Visibility efforts are paying off with more classes than ever joining our program without us having direct contact with them.

Supporting students and instructors

A key facet of the Year of Science was to build the support structure needed to ensure we could bring thousands more students to Wikipedia successfully. We did that through two major initiatives: technical tools and handouts.

Developing technical tools

In order to support the larger number of students editing Wikipedia during the Year of Science, we invested a significant amount of money in the 2015/16 fiscal year in developing our Dashboard course management software. To do this, we hired a very good but expensive web development agency, WINTR, to do the design, development, implementation, and management thereof our software projects. We created a modular online training platform, an instructor survey tool, and Wiki Ed staff-facing tools that enabled us to see what students are doing more easily and alerted us to potential issues, so we can intervene when problems crop up. Each of these were key in enabling us to support more classes. Product Manager Sage Ross spent the 2015/16 fiscal year working closely with WINTR, to the point that when the 2016/17 fiscal year rolled around, he had enough technical skills himself to both manage the day-to-day technical work as well as to complete a wide range of development tasks himself. This significant investment in the Dashboard platform was crucial to how we were able to successfully scale the program up without adding staff time.

Creating discipline-specific handouts

A key project we undertook for the Year of Science was to create a series of discipline-specific handouts in support of the Year of Science. With the support of Wikipedia Content Experts Ian Ramjohn and Adam Hyland, we created handouts on biographies, chemistry, ecology, environmental sciences, genes and proteins, linguistics, political science, and species during the preparation phase and during the actual Year of Science (they joined already popular guides on psychology and medicine topics). These handouts were key for giving student editors what they needed to know to contribute effectively to those particular article topics: how would you structure types of articles, what are the appropriate types of sources, etc. We had few incidents in science classes during the Year of Science, which points to the importance of these handouts.

Learnings: Supporting students and instructors

Early investments in our Dashboard software, such as modular online trainings and alerts about content for Wiki Ed staff enabled us to support thousands more students with the same staff support.
Discipline specific handouts were popular with students and seemed to head off incidents.

Retaining instructors

While Wiki Ed doesn't put effort into retaining student editors after the end of the term, we do put significant effort into retaining instructors. After all, a retained instructor will continue bringing new students and filling more content gaps term after term, with little additional input needed from Wiki Ed. What we did during the Year of Science was in alignment with what we were doing before in terms of instructor retention, but with a focus on a more scalable approach and science-specific content.

Classroom Program Manager Helaine Blumenthal did so by reaching out to instructors via targeted emails to raise awareness that they were participating in the Year of Science. We tried to create a community of instructors interested in improving science content on Wikipedia, conveying that their students were doing a civic good.

Our first attempt was a webinar on science communication with Wikipedia assignments, co-hosted with Louisiana State University's Becky Carmichael. While the content was well-received and feedback from attendees was overwhelmingly positive, it took a lot of time to develop the material from the webinar, and attendance was low. A post-webinar survey revealed that what instructors valued most was the opportunity to ask questions. In response to this, we instituted “office hours” where staff are available to answer any questions via video chat for an hour twice each month.

The communications elements of the Year of Science retention were challenging. Our sense was that instructors cared a lot more about their own reasons for teaching with Wikipedia, and didn't particularly care that they were part of the initiative. It was difficult to convey a sense of urgency for why we were doing it now, but we also wanted instructors to continue teaching with Wikipedia in their science classes after the initiative was over, so we needed to walk the fine line between generating enthusiasm for the initiative and generating enthusiasm that ended at the completion of the initiative.

In retrospect, we should have created more programmatic activities for instructors and students, such as Year of Science awards or contests, or perhaps had some sort of wrap-up event for instructors. We should have played down the “year of” aspect and better encouraged the idea that it was ongoing work.

There is an open question of whether participants who taught for the first time during the Year of Science will be retained at the same level as others; in other words, was participation in the campaign a motivator that goes away after the conclusion of the campaign? It will take several more terms to be able to get enough data to really measure this.

Learnings: Retaining instructors

“Office hours” were more useful than a structured thematic webinar.
We never got the sense instructors really cared that they were participating in the campaign; they were motivated by the individual outcomes for their class and students.
More programming around awards or contests could have helped affiliation with the campaign.

Visiting Scholars Program

The focus of the Visiting Scholars program is quality: Individual Wikipedians are connected to individual universities, and work on developing articles in a particular topic area.

In recruiting program participants for the Year of Science, Jami and Samantha focused on the Classroom Program because we already have learned how to target disciplines and recruit instructors into that program. Channeling recruitment efforts into the Classroom Program meant we had little time to devote to recruiting hosts for Visiting Scholars. Interested hosts occasionally discover logistical limitations in offering Wikipedia editors access to library resources, making it a time-consuming challenge to pursue Visiting Scholar hosts. The outreach team believed given the time crunch of the Year of Science, it was more important to focus their limited energy on recruiting for the Classroom Program.

Community Engagement Manager Ryan McGrady also had challenges filling one of our open Visiting Scholar positions related to the Year of Science, at the University of North Carolina. UNC is seeking a Visiting Scholar to improve articles about clinical psychological science, with a possible emphasis on evidence-based assessment (especially open access/public domain tools, rather than commercial products). We learned this extremely narrow interest area was very hard to recruit for, and despite opening the position in July 2016, we still have not filled it nearly a year later. We also think one challenge in recruitment for the sciences in particular is that some Wikipedians in the sciences may not need access to sources, but are instead interested in career opportunities. We believe providing some sort of honorarium or stipend for Visiting Scholar work could help with recruitment woes.

Our one successful Visiting Scholar story from the Year of Science is Wikipedian Barbara Page, who worked at the University of Pittsburgh. Barbara worked on articles related to women's health, creating dozens of articles and improving hundreds more. Barbara's impact can be seen in articles on neonatal infections, rape, breastfeeding, and STIs. Barbara's work was key to providing well-sourced, reliable information about topics many people are seeking information on.

Learnings: Visiting Scholars Program

Our one science related Visiting Scholar made a huge impact, improving critical articles related to women's health.
Too narrow of Visiting Scholar requests made filling positions difficult.
Wikipedians in the sciences in the United States often already have access to sources through their jobs, as they often work in a science field.
We want to explore offering some form of stipend or honorarium to Visiting Scholars to see if we can increase the impact of the program.

Other projects

Playlist

In late 2015, Executive Director Frank Schulenburg and Director of Programs LiAnna Davis met with the Wikimedia Foundation's communications team about supporting the Year of Science. They were particularly interested in creating a social media friendly tool that would be a “Playlist” of favorite Wikipedia articles, as part of Wikipedia's 15th birthday, which coincided with the Year of Science launch. Wikimedia Foundation communications staff encouraged Wiki Ed to develop this tool, and verbally agreed to partner with Wiki Ed in promoting the tool. Wiki Ed spent a significant portion of our development resources for nearly two months building the tool, only to have the Wikimedia Foundation back out a few days before launch. Without the support expected from WMF, the tool did not become the social hit we were hoping for.

We learned a lot from this experience. From the software development perspective, we would have spent less time polishing the tool before we validated its potential for success. From a project management perspective, we learned we needed to confirm the agreement with the Wikimedia Foundation communications staff in writing; verbal commitments were too easy for them to back away from. From a fundraising perspective, we learned that having exciting add-ons to grant proposals can be key, but we need to be sure we're highlighting that they are pilots so expectations are set properly.

Simons edit-a-thons

At several of the Year of Science conferences we attended to recruit science instructors, we joined the Simons Foundation in running edit-a-thons to teach academics how to contribute to Wikipedia. We hoped these would serve as a strong foundation to recruit attendees into the Classroom Program, which we know amplifies the impact they have on Wikipedia. However, we did not find a lot of success in recruiting instructors and found, instead, that attendees were interested in developing their own skill rather than thinking about their role as a teacher. We are glad to have experimented with these, since people who are not Wikipedia editors are so motivated by edit-a-thons, but now we know not to spend our limited resources on supporting edit-a-thons.

Community projects

Prior to the Year of Science, Ryan did a series of interviews with community members active in the sciences areas to try to figure out ways to make the Year of Science bigger than just something Wiki Ed ran. Despite doing some initial work, and creating an on-wiki portal, we never really pushed any community projects. This had been initially scoped, but was one of the things we cut back on when funding didn't come through right away. We were hoping that something might organically come together, but we understandably didn't see much action. Wikipedian Lane Rasberry outlined in detail many of his takeaways of the Year of Science, and we agree with his conclusions.

We focused our limited energy on piggybacking on existing community events. We supported the work of the Women in Red group to improve biographies of women scientists, and we sponsored a science side competition in the WikiCup.

Learnings: Other projects

Exciting technical add-ons like the Playlist can be good for closing funding deals, as long as they are explained as pilots to funders.
Scope small projects, get partnerships in writing, and validate early in the process.
Edit-a-thons were not a productive mechanism for recruiting instructors into our Classroom Program.
More engagement with community is needed to actually have impact on community projects.

Organizational impact

Fundraising

A key facet of the Year of Science was that we believed it could be an opportunity to raise significant funds in support of our organization. In particular, the STEM funding scene is quite active, and we wanted to align our programatic impact to where funders are. Our approach to fundraising was to pursue large foundation grants and smaller corporate gifts, concurrently with raising new, general support for Wiki Ed.

Senior Manager of Development Tom Porter started raising money for the Year of Science in mid-2015, building relationships, having one-on-one meetings with funders, hosting events and small group luncheons, and a lot of cold calls, letters of interest, and social media. The outcome was that we were able to raise a total of $1.38 million in support of our science work. While this is a significant accomplishment and provided truly crucial funds for our organization, it fell short of our goal for the initiative, which was $1.8 million.

Additionally, timing became an issue: Similar to the learnings from recruiting faculty, a big learning was that we needed to start the cultivation of relationships with potential donors significantly earlier. We ended up having to eliminate or delay spending in the budget after money hadn't come in when we needed it to. The cultivation cycle is long in fundraising: We had 27 individual points of communication with our contact at Google before Google provided us $500,000 in support of the initiative. Funders put together their budgets in advance, and being able to be on their radar months or years prior to when we needed the money would have helped tremendously in this case.

We also learned that while there are a lot of organizations funding STEM work, there are also a lot of organizations seeking STEM funding. Many of the STEM funders are exclusively targeting K–12 projects, as well, and our argument that improving Wikipedia content in the sciences helps K–12 students wasn't enough to differentiate us in the competitive field. An ongoing challenge we have is that some new funders invite us to meetings because they're curious about Wikipedia or their own organization's Wikipedia articles, and see this as an opportunity for them to learn, while having no interest or ability to give us funds.

Initially, several funders suggested we should connect with staff at the White House's Office of Science and Technology Policy. Frank and Tom were able to secure a meeting at the White House with OSTP staff, who expressed enthusiasm for the initiative, but the only outcome was a reference to Wiki Ed in a blog post about NASA mathematician Katherine Johnson receiving the Medal of Freedom. Similarly, Tom attended a Philanthropy New York event to highlight the work of the Year of Science; while these raised visibility, they did not result in funding. Encouragement from the White House is obviously great, but in future campaigns, we'd want to figure out how to best translate this into outcomes for us.

Another area that proved challenging for us was the lack of a feedback mechanism when our proposals were rejected. Funders generally didn't express any particular reason for rejecting our proposal, which made it hard for us to learn and improve for future iterations. When we did have a successful proposal, such as with the Simons Foundation, we were able to then establish a good enough relationship to determine what it was about our proposal they particularly liked.

One very important learning was that we had a good argument for the corporate funders in the employability skills area. Many STEM funders are concerned about their workforce development, and are actively funding programs that increase the critical thinking, writing, research, and collaboration skills of their future workforce; communicating science to the general public, as we teach through writing Wikipedia articles, was a key successful selling point for many funders.

Learnings: Fundraising

Cultivation cycle means we needed more time from starting to cultivate new funders to the point where we could get the money in our account.
Unsolicited proposals didn't work as well as ones where we'd cultivated a relationship with the program officer to the point where they told us what to ask for in a written proposal. With more time, we could've cultivated those relationships with more funders earlier, to be able to get the money when we needed it.
Employability skills and workforce development are key for corporate funders, and we have a real opportunity to grow our funding in this area because our program aligns to that area so well.
We haven't figured out yet how to best capitalize on awareness/exposure connections.

Communications

Media outreach

We anticipated that the Year of Science would be a big media opportunity, and to a certain extent, it was — but not as much as we'd hoped. Journalists were not particularly excited about the kickoff (“'Year-of' campaigns aren't newsworthy” was a common refrain we heard throughout the process). We reached the pinnacle of our news coverage in the middle of the campaign — nobody wanted to write about it when it kicked off, nor when it wrapped up, but our biggest pieces were in the Washington Post and the LA Times, which both wrote stories midway through the campaign and got widespread news wire distribution.

The biggest drivers of coverage were our partners and participants: journalists are naturally suspicious of self-driven coverage of organizations, and when the information focused on our partners or our program participants too, we had much more success.

Another learning for us is that edit-a-thons are perplexingly newsworthy. While the Simons Year of Science edit-a-thons didn't produce much in the way of content, they generated a lot of news coverage. We think the fact that it's an event, with a short duration, and in person, makes edit-a-thons a more digestible news feature, even though the impact of our classes is significantly greater.

Blog posts

We maintained consistent traffic to our blog posts about science topics. Communications Manager Eryk Salvaggio did an excellent job telling the story of the Year of Science in a series of well-read blog posts. In particular, his work on creating longform pieces about topics related to science pedagogy were particularly powerful in bringing traffic to our site.

Our Year of Science landing page got about 5% of our overall web traffic in 2016, second only to our home page. It served as a meaningful introduction to our project for a variety of audiences.

Social media

While we had a few social media wins (a Facebook post about a student who wrote the Geobiology article reached more than 15,000 people, easily our biggest social media impact), we struggled to convert that to actual traffic to our website or classes. Instead, social media seemed to be most effective at ongoing engagement with existing faculty and to encourage traffic to our booth at conferences. In particular, photos of students we post to Twitter get the most impressions.

Learnings: Communications

Year-of campaigns aren't newsworthy in themselves.
Middle of the campaign is the best time to get coverage.
Partners and participants are the best at driving coverage.
Edit-a-thons are perplexingly newsworthy.
Good longform science-related blog posts drew significant traffic to our website.
Social media is most effective at engagement with existing faculty and to draw traffic to our booth at conferences.

Goals and outcomes

We exceeded all of our goals:

Measurement	Goal	Actual
Articles edited	5,000	5,710
Amount of content added	4.5 million words	4.99 million words
Images added to Commons	1,000	1,970
Students participating	4,500	6,336
Biographies of women scientists	100	150

While this qualifies as a successful initiative, there are some learnings. In particular, our past course work had indicated that each student added on average 1,000 words of content to Wikipedia; this dropped during the Year of Science. We believe this is because we brought on board more larger classes; we directed those classes to smaller assignments.

We were also really impressed with the quality of work among all of our students. In particular, students working on the life sciences topic areas really gravitated toward the assignment and created some significant content. We think this is because biology classes in general already had more writing assignments, and so students were already well equipped to write about biology topics on Wikipedia. There were also a lot of important content gaps on Wikipedia students were able to easily fill.

The amount of content students added is impressive: nearly 5 million words fills 3.5 full volumes of the last print edition of Encyclopedia Britannica. And the work has had a lasting impact: the articles improved through the Year of Science have been read more than 333 million times within one month of their edits. Special credit for this large number goes to our one Visiting Scholar in the sciences, Barbara Page, whose work accounts for a third of that total.

While we easily met the image goal numerically, we weren't happy with the outcomes. We wanted students to either take their own photos to illustrate scientific topics, find open licensed imagery, or create diagrams to explain concepts. While many students did this well, others struggled to understand the concept of open licenses and uploaded images they'd found online. Interventions we made in adding training content about images didn't seem to help; in an effort to not overwhelm Wikimedia Commons with mis-attributed imagery, we de-emphasized this element of the program.

Finally, we were really impressed with the quality of biographies of women scientists. This element of the program allowed a handful of humanities instructors who wanted to participate to join. We specifically want to call out a large introductory geology class, which worked to improve more than 50 biographies of women geologists.

While we ended up meeting the numeric goals, we were not always confident we would achieve them. In particular, we set these goals with a planned $1.8 million budget, and promised them in initial grant proposals to funders who committed to funding part of that. But when we didn't raise the full amount, we still needed to meet the goals. In the future, we wish we had set a tiered amount (so a funder who contributed a certain amount would be sponsoring a certain amount of content added); this eliminates the challenge of needing to meet large goals while not having the money we planned to have to accomplish it. It speaks volumes about the Wiki Ed programs team that they were able to pull off and exceed the numeric goals despite a reduced budget.

Learnings: Goals and outcomes

Students, particularly in the life sciences areas, produced really great content in the sciences on Wikipedia.
The Year of Science produced nearly 5 million words of content, which has been read by more than 333 million people.
We need to rethink how we approach encouraging students to add quality images to Commons.
In future initiatives, we will tie funding levels to outcomes, rather than set an initiative-wide goal that we promise to all funders, regardless of how much they give.

Conclusion

In our Annual Plan for 2015/16, we wrote: “The Wikipedia Year of Science is...the fulfillment of the board and the senior management's desire of setting an audacious and aspirational long-term goal for our organization. Embarking on a large-scale initiative will bring focus to our work, encourage teamwork, and increase fulfillment and pride among staff. At the same time, a larger and more visible project like the Wikipedia Year of Science will offer new opportunities for fundraising and for improving our organization's image in the public.”

In a number of these areas, we had great success:

The initiative brought focus to our work.
It encouraged teamwork among staff members.
We met all of our numeric goals.
We had a visibility boost that paid off in terms of new classes.
Our Dashboard development work enabled us to scale.
Our support structure was strong enough that we had no major science incidents.
Our training content and discipline specific handout gave students the information they needed.
Overall, we absolutely created a scalable model.

But we had challenges in other areas:

We wanted to see more community engagement than what actually happened. We would need to invest significantly more of Wiki Ed time and resources to make this happen if we wanted to do another initiative like this.
Projects like the Playlist and Simons Edit-a-thons didn't produce what we wanted.
We backed off the image goals after realizing students struggle to understand copyright issues.
We wanted the Year of Science to be an initiative where we could do more experimental programs and expert engagement; these parts of the plan ended up getting cut when funding was delayed.
We hoped the initiative would drive more partnerships, but we had to sacrifice partnership development for a focus on recruiting classes.
We never quite generated the enthusiasm for the initiative among instructors and students; instead, they remained focused on their own part without a sense of the bigger picture.
The rapid scaling put a lot of pressure on staff; whether we met the “fulfillment and pride” elements for staff are debatable.

A common theme running throughout all the sections is more advance time: we need an appropriate time frame between receiving the funding commitments and when the work starts.

Taking on the Year of Science gave us a much better idea of what it would take to do any large-scale initiative in the future. We developed much better processes and procedures internally so that we could scale our impact. In particular, our technical support from the Dashboard and Salesforce made a huge impact on nearly everyone's work.

Finally, we would be remiss without mentioning that our experienced staff was crucial for the success of this project. The core of staff working on the Year of Science all joined the organization prior to the first development of the project in mid-2015 and stuck with the organization throughout the initiative. The dedication and expertise of all staff members was crucial for the success of this project. We truly couldn't have pulled it off without a lot of hard work from everyone on the team.

Appendix

Appendix 1: List of conferences we attended as part of the Year of Science

Society for Marine Mammalogy Conference, December 2015
Linguistic Society of America Annual Meeting, January 2016
American Association for the Advancement of Science Annual Meeting, February 2016
Association of American Geographers Annual Meeting, March 2016
Experimental Biology, April 2016
American Astronomical Society Meeting, June 2016
American Society of Plant Biologists Annual Meeting, July 2016
The Allied Genetics Conference, July 2016
Botany 2016, August 2016
Joint Statistics Meeting, August 2016
MathFest, August 2016
Ecological Society of America Conference, August 2016
American Chemical Society Fall Meeting, August 2016
American Anthropological Association Annual Meeting, November 2016
American Geophysical Union Annual Meeting, December 2016

Appendix 2: List of media about the Year of Science

Wikipedia and the Year of Science Craig Newmark, craigconnects (January 14).
Wiki Education says 2016 is the Wikipedia Year of Science John Timmer, Ars Technica (January 22)
Wikipedia Year of Science 2016 Laura Guertin, GeoEd Trek (January 27)
Science Students Are Writing Wikipedia Articles Instead of Term Papers Meghan Neal, Motherboard (Vice) (February 11)
Os estudantes de ciência que publicam e revisam artigos da Wikipédia Meghan Neal, Folha De San Paulo (Brazil) (February 15)
The Dos and Don’ts of Wikipedia Editing in the Undergraduate Psychology Classroom Christina Shane-Simpson and Patricia J. Brooks, Association for Psychological Science (February 16)
Edit-a-Thon Seeks to Fix Wikipedia’s Gaps in Science Juan David Romero, American Association for the Advancement of Science (February 16)
Term papers? These science students write Wikipedia pages instead of term papers Max Lewontin, Christian Science Monitor (February 16)
AAAS hosts Wikipedia Edit-a-Thon Drew Costley, AAAS Member Central (February 23)
Cornell takes big red pen to Wikipedia life sciences content Amruta Byatnal, The Cornell Chronicle (March 22)
Wikipedia Year of Science: An open opportunity for participation PLOS (April 14)
Edit Wikipedia at Plant Biology 2016 Katie Engen, ASPB Plant Science Today (May 13)
OU History of Science Students Write More than Term Papers University of Oklahoma Public Affairs. (May 27)
In the Wikipedia Year of Science, core USP updated entries on brain theory Journal da USP, Brazil (Portuguese) (May 30)
The Year of Science is coming to Plant Biology 2016, Katie Engen, ASPB Plant Science Today (May 31)
Wikipedia Year of Science 2016 Midtown Tribune (May 31)
Students improve Wikipedia’s science, build skills and benefit all Allison Ong, Daily Bruin (June 2)
Wikipedia’s Year of Science 2016 helps students get to grips with science communication AIMS Team, United Nations (June 5)
ASM hosts Wikipedia Edit-a-thon as part of the Year of Science mBiosphere, American Society for Microbiology blog. (June 10)
What science frontiers have to do with Wikipedia Celio Costa Filho, Are Ede Educa (Brazil) (Portuguese) (June 13)
Wikipedia “Year of Science” Edit‐a‐thon Meredith Rawls, Astrobites (June 16)
The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, Washington Post (June 20)
1. Why some college professors are telling students to use Wikipedia for class Caitlin Dewey, LA Times (repub) (June 20)
2. Why some college professors are telling students to use Wikipedia for class Caitlin Dewey, Orlando Sentinel (June 20)
3. Why Wikipedia belongs in schools Caitlin Dewey, NY Post (repub) (June 21)
4. Colleges rethink their approach to Wikipedia Caitlin Dewey, Denver Post (repub) (June 22)
5. Why some professors are telling students to use Wikipedia Caitlin Dewey, Chicago Daily Herald (repub) (June 26).
6. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, State Journal Register (Illinois) (Repub) (June 27)
7. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, Mineral Daily News Tribune, (June 27).
8. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, Jackson Newspapers, South Carolina (June 27)
9. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, Star Courier, Illinois (June 27)
10. Editing Wikipedia Entries Is Becoming A Class Assignment In Colleges Across The U.S. Caitlin Dewey, Hartford Courant, June 27
11. College student edits boost Wikipedia data credibility Caitlin Dewey, Northwest Arkansas Democrat Gazette (June 28)
12. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, The Evening Tribune, New York (June 28)
13. The surprising reason some college professors are telling students to use Wikipedia for class Caitlin Dewey, Gennesee County Express, New York (June 28)
14. Some college professors surprisingly are telling students to use Wikipedia Caitlin Dewey, Star Tribune, Minneapolis (July 8)
Help Diversify the Largest Encyclopedia in the World through Wikipedia Assignments Jessica McCullough, Engage: Teaching with Technology at Connecticut College (blog) (June 28)
Wikipedia project spurs research, service learning North Dakota State University (July 11)
Students take on role of Wikipedia editors George St. Martin, News @ Northeastern (University) (August 1)
Чому Вікіпедія важлива для жінок у науці Eryk Salvaggio, translated into Ukranian by Vira Motorko for Wikimedia Ukraine. (August 1)
When the professor says it’s OK to use Wikipedia Janelle Nanos, Boston Globe (scroll down for story) (August 2)
O ano da ciência se propaga no Brasil Eryk Salvaggio, translated into Portuguese by Victor Barcellos for Ciencia Aberta (August 8)
Midwest Political Science Association (MPSA) Newsletter (August 18)
Volunteer Expands Pitt’s Reach, One Wikipedia Citation at a Time Sharon S. Blake, Pitt Chronicle (August 22)
5 razões pelas quais tarefas na Wikipédia podem melhorar habilidades de comunicar ciência Eryk Salvaggio, translated into Portuguese by David Alves for the Neuromat Blog (August 23)
College students take to Wikipedia to rewrite the wrongs of Internet science Rosanna Xia, LA Times (September 20)
1. Professors once disdained Wikipedia. Now they assign their students to rewrite it Miami Herald (reprint of LA Times, October 2)
2. Professors once disdained Wikipedia, but now assign their science students to rewrite it Virgin Island Daily News (reprint of LA Times, October 3)
3. Professors once disdained Wikipedia, but now assign their science students to rewrite it Texarkana Gazette (reprint of LA Times, October 3)
4. Professors once disdained Wikipedia, but now assign their science students to rewrite it The Bulletin, Bend OR (reprint of LA Times, October 3)
5. Science profs assign students to rewrite Wikipedia entries Leader-Telegram (reprint of LA Times, October 9)
Why Are These Giant Funders Giving Money to Make Wikipedia Better?, Tate Williams, Inside Philanthropy (October 4)
Should I trust Wikipedia with my health?, NPR, November 8
How do you know which medical information to trust?, KQED, November 28
Wikipedia Wunderkind: Barbara Page helps readers of the online encyclopedia understand women’s health, among other topics, Marylynne Pitz, Pittsburgh Post-Gazette (December 15, 2016)
How Undergraduates Can Expand Wikipedia, Advance Chemistry, Jami Mathewson, In Chemistry (December 15, 2016)
Wikipedia Year in Science incorporated into science writing assignment, Susan Cannon, U of Calgary News, (December 21, 2016)
Year of Science 2016 – a new model for Wikipedia outreach, Lane Rasberry (January 7, 2017)
Rutgers undergraduates improve public access to scientific knowledge through Wikipedia, Rutgers (January 2017)
Students Bolster Wikipedia’s Pet Nutrition Content with Science, Stephanie Craig, University of Guelph (February 1, 2017)
Students bolster Wikipedia pet nutrition content, Feedstuffs (February 3, 2017)