Learning and Evaluation/Evaluation reports/2013/Wikipedia Education Program
Part of the Wikimedia Resource Center
Learning and Evaluation
This is the Program Evaluation page for the Wikipedia Education Program. It currently contains information based on data collected in late 2013 and will be updated on a regular basis. For additional information about this first round of evaluation, please see the overview page. This page reports data from six program leaders, detailing a total of seven Wikipedia Education Program implementations.
We asked for program leaders to report on their programming from the 2012-2013 academic year. Still, the length of time varies in the implementation of these Wikipedia Education Program (WEP) activities, as does the tenure of the program leaders involved in implementing these activities. A Wikipedia Education Program 'implementation' implies the range of activities executed by a WEP program leader for any given length of time.
- Three program leaders reported on implementations that are on-going. At the time of reporting, the first of these had been running for 36 months, and two had been running 21 months.
- Two program leaders reported on implementations in their own courses: one reported a single course implementation lasting 9 months (two terms), the other reported two implementations, one lasting 2 months, the other lasting two weeks.
- One program leader contributed report data for a pilot education program which lasted 2 months.
Each of the seven implementations reported are included in the report. No additional data was mined for this evaluation. Program leaders who responded to the pilot survey about the Wikipedia Education Program were primarily associated with chapters who were running the implementations, with two individuals not directly associated with chapters. You can learn more about response rates and limitations here.
Key lessons include:
- Wikipedia Education Programs focus on content production and improvement, rather than on editor retention. The programs surveyed maintain a steady flow of new student editors through retaining instructors.
- Program leaders have five priority goals centered around increasing contributions to, quality of, and respect for Wikimedia projects.
- Program implementation reported cost a total of almost $8,000 USD in total, and $275 each week to implement.
- Most Wikipedia Education Program implementations don't use donated resources, and those that do are primarily volunteer driven. Those that do, reported prizes/give-aways as the most popular.
- Program leaders reported participation rates ranging from 25 to 2,372, and program implementations lasted from two weeks to 21 months with an average of 37.5 weeks.
- The average Wikipedia Education Program implementation invests a total of $67 US and 3 hours into recruiting each new editor participant.
- The average Wikipedia Education Program implementation produces about 120 pages of content each week. The average program participant adds about a quarter-page of text content to Wikipedia and creates or improves six wiki pages each week.
- The average Wikipedia Education Program implementation reported invests about $28 USD per page of content created.
- For the seven reported Wikipedia Education Program implementations, almost 3,000 different Wikimedia pages were created or improved.
- Of the 3,334 new editors who participated in the reported Wikipedia Education Program implementations, 36 (1.2%) were "active"[1] three months after the program ended; 33 (1.1%) were "active" six months after the programs ended.
- The Wikipedia Education Program implementations range from small, volunteer-run programs with little to no budget to large-scale programs with significant investments of resources (staff and volunteer time and money). All reported implementations succeeded in recruiting new student editors who contributed to Wikimedia projects; more data are needed to support comparison or conclusions across program models.
- Documentation for replication is one of the Wikipedia Education Program's strengths. The majority of program leaders indicated having experience in leading education program activities and reported that they produce and share learning resources, including documentation, materials, and how-to's to aid others in implementing their own versions of the program.
Planning an education program? Check out some process, tracking, and reporting tools in our portal and find some helpful tips and links on this resource page
Program basics and history
[edit]
The Wikipedia Education Program has one main mission: to teach students how to edit Wikipedia and have those students contribute to Wikipedia as part of their classroom assignments. The basic concept is that students write Wikipedia articles instead of writing a traditional term paper. They receive support and help through a network of volunteer "Wikipedia Ambassadors" as well as through a variety of printed and online materials (including an online training that was developed in 2012). With a focus on content improvement, instead of editor retention, the program produces large amounts of content each term around a variety of subjects, ranging from women's history to biology, and working on language skills through translating Wikipedia articles. In addition to adding text, some students have also been assigned to add graphics, charts, photos, and videos to Wikipedia. Student contributions can be tracked and coordinated through a MediaWiki extension that has been activated on several Wikipedia language versions.
Although professors had been using Wikipedia as a teaching tool since 2003, the Education Program, as it is now known, grew out of a pilot project, the Public Policy Initiative, which was run by the Wikimedia Foundation in 2010-11 to determine the feasibility of systematically improving content in a specific content area. In the pilot, professors in public policy programs in the United States assigned their students the task of improving Wikipedia articles on English Wikipedia. The Wikimedia Foundation provided lesson plans, and also built a volunteer community of Campus Ambassadors and Online Ambassadors to mentor the students. When the Public Policy Initiative wrapped up, and it reported having a significant impact on content production, volunteers and chapters around the world adopted the concept. As of now, Wikipedia Education Program branches have been established formally in 20 countries, with educational activities underway in an additional 40 countries.
Data report
[edit]Response rates and data quality/limitations
[edit]Six program leaders provided data about seven Wikipedia Education Program implementations from 2012 and 2013. These events ranged from lasting two weeks to 86 weeks. [2] Three of the program implementations were on-going, one was a pilot program, one was oriented around an on-going specific class, and the last two were individual one-time classes.
Since reported program implementation durations were so varied, it was like comparing apples and oranges. To minimize the effects of such variation in making comparisons, we used weekly averages for outputs and outcomes to draw information across these different implementations, due to the varying program length and reporting windows. Still, because of the differences in implementation status and potentially different programming goals (i.e., recruitment of new editors, recruitment of education partners, etc), comparisons across program implementations should be done with caution.
Note: Two program implementations included in this report - the US/Canada and the Arab world programs - were managed (at the time of implementation) by Wikimedia Foundation education staff. In order to maintain the neutrality of this report, current education staff members as well as former education staff members were not involved in any data analysis for this report.
Report on the submitted data
[edit]Priority goals
[edit]- According to respondents, their Wikipedia education program implementations have five priority goals.
We asked program leaders to select their priority goals for their Wikipedia Education Program implementations. We provided 18 priority goals with an additional 19th option to report on "other" goals. They could select as many or as few as they saw fit. All six of the program leaders who reported on the Wikipedia Education Program selected priority goals for their workshops. They selected between five and 15 priority goals for each of their events.[3] Our team noted five stand-out goals that appeared as priorities among the reporting program leaders (see table below):
Inputs
[edit]- In order to learn more about the inputs that went into planning Wikipedia Education Program implementations, we asked program leaders to report on
- The budget that went into planning and implementing the Wikipedia Education Program
- The hours that went into planning and implementing the Wikipedia Education Program
- Any donations that they might have received for the event: a venue, equipment, food, drink, giveaways, etc.
Budget
[edit]- The majority of program leaders reported budgets. The average cost of the reported programs was a total of almost $8,000 USD, and $275 per week to implement.
Six out of seven Wikipedia Education Program reports (86%) included budgets. Budgets ranged from $250 USD to $264,122 USD. The program average was $7,775 USD.[4]
We also looked at budgets based on how many weeks program implementations might last. Looking at these budgets by weeks of programming, weekly budgets ranged from $2.91 per week for an on-going program that is primarily volunteer-based and reaching 300 participants since its inception to $5,079.27 per week for an on-going national program reaching 2,372 participants annually. On average, Wikipedia Education Program implementers reported cost $275 each week to implement.[5]
We were also able to determine the cost per participant per week. Weekly budgets towards each participant ranged from $0.01 USD to $8.33 USD. The average budget per participant in the reported Wikipedia Education Program implementations was $2.75 USD per participant.[6]
Hours
[edit]- Most program leaders were able to report how much volunteer time went into implementing their Wikipedia Education Program implementations. Only a minority was able to report staff hours.
Staff and volunteers put the following time into implementing edit-a-thons, according to respondents:
- Six program reports (86%) included volunteer hours, which ranged from 3.5 to 269 hours per week with an average of 24.[7]
- Two program reports (29%) included staff hours, which ranged from 72 to 77 hours per week with an average of 74.5 hours.[8]
- Total hours (staff and volunteer hours combined) ranged from 3.5 to 346 each week, with an average of 24 hours each week.[9]
We also calculated how much time staff and volunteers spent per participant on a weekly basis. Weekly hours ranged from .6 to 36.6 minutes each week, per participant. On average, 25.2 minutes went toward supporting each participant each week.[10]
Donated resources
[edit]- Most Wikipedia Education Program implementations don't use donated resources, and those that do are primarily volunteer driven. Those that do, reported prizes/give-aways as the most popular.
Just under half (43%) of program leaders reported using donated resources, all of which were reports from volunteer-run programs and the pilot program. None of the on-going established program implementations reported donated resources. Prizes/give-aways were the most commonly donated resources at 43% (see Graph 1).
Although program leaders did not report "meeting space" as a donated resource, instructor classrooms are an inherent part of the Wikipedia Education Program and the question will need to be better clarified for future reporting.
Outputs
[edit]- We also asked about two outputs in this section
- How many weeks of programming are included in the report?
- How many people participated in the program during the reporting duration?
Participation and program length
[edit]- The average Wikipedia Education Program has 71 participants and is eight and a half months long.
Program leaders reported participation rates ranging from 25 to 2,372. The average program had 71 participants.[11] The implementations reported lasted from two weeks to 21 months. The average Wikipedia Education Program lasts 37.5 weeks (see Graph 2).[12]
-
Graph 2: Programming duration reported. This pie chart shows that Wikipedia Education Program reports varied in their length of programming time. The slight majority (57%) reported on programming that lasted at least 6 months.
- The average Wikipedia Education Program invests, on average, a total of $67 USD and 3 hours into recruiting each new editor participant.
Since we had budget, new participant counts, and input hours for six out of seven reported implementations, we were able to learn how much money and time went into each new participant in each program implementation. The variability was high. The total input costs for recruiting each new editor ranged from $0.83 USD to $370.15 USD with an average of $66.67 USD.[13] The amount of time invested in recruiting each new participant ranged from no reported staff or volunteer time to 29.4 hours per participant, with an average of 3 hours (see Graph 3).[14]
-
Graph 3: Dollars and hours to participants. This bar graph depicts how many dollars and hours were invested per participant for each of the reported program implementations. Note: These bars depict full spending, not spending weighted by weeks of programming (see next).
In order to control for the high variability in reporting time periods, we took the budget and hour inputs, and the participant outputs, and assessed them in terms of weeks of programming, instead of the entire program period (see Graph 3). Per week, these programs invested a range of $0.01 USD to $8.33 USD per participant with an average of $2.45 USD.[15] Regarding hours, program implementations invested from 6 to 48 minutes per participant. On average, time investment was 25 minutes per participant (see Graph 4).[16]
We learned that two program implementations made the lowest investments but reached the largest participant base. One was mostly volunteer-driven while the other was an on-going program with staff with the most implementation classrooms and largest student pipeline.
-
Graph 4: Dollars and hours to participants. This bubble graph depicts average weekly inputs in terms of dollars and hours to participants averaged over the time period program implementations reported. The size of the bubble, and corresponding number they are labeled with, show how many new account creators contributed to the projects through each reported program implementation. The location of the bubble in relation to the x- and y-axes illustrates the weekly average US Dollars and Hours input into each program, per participant, for its reported duration. Here we see that the the two bubbles lowest along x- and y-axes demonstrate the least inputs per new account creator. Other bubbles on the graph illustrate the variability and potential high inputs expended per participant in the other reported program implementations.
Outcomes
[edit]We collected three types of content production outcomes:
- Three program implementations (43%) reported the number of characters added
- Six program implementations (86%) reported the number of pages created or improved
- Six program implementations (86%) reported the number of media uploaded
- The average Wikipedia Education Program produces about 120 pages of content each week. The average program participant adds about a quarter-page of text content to Wikipedia in a week.
Three program implementations (43%) reported how many characters their participants added to Wikipedia. We calculated how many characters fit on a letter size piece of paper. We processed the characters added based on how many pages of text were added.
The three program implementations (43%) reported numbers ranging from 418.5 pages to 10,147.87 pages. The average was 120.27 pages of text added each week.[18] We also calculated how many pages each participant added per week. Rates ranged from 0.08 to 0.26 pages per week. The average participant added 0.23 pages each week.[19]
- The average Wikipedia Education Program creates or improves six wiki pages each week.
Six program implementations (86%) reported how many pages their program participants created or improved on wiki. Reports ranged from two to 26 pages created or improved each week. The average program creates or improves six pages each week.[20]
In terms of pages created or improved, six program implementations (86%) reported the number of pages that were created or improved by program participants on average each week of programming. Reports ranged from 2 to 26 pages created or improved per week of programming with an average of 6. [21]
- The average Wikipedia Education Program reported invests about $28 USD per each page of content created during the length of the program.
We were able to determine how much budget went into the creation of text on wiki during the programs by dividing the dollars invested by the number of pages of text added. Program implementations spent between $26.03 USD to $35.84 USD for each page of text added. The average was $27.70 USD [22](see Graph 5). Although we observed that the most productive of the three program implementations with reported data had the highest budget per participant, we cannot draw any conclusions without many more reports of program data (see Graph 6).
-
Graph 5: Dollars to pages. This box plot depicts the distribution of how many dollars were invested for each printed page added during the event. As illustrated, by the long vertical line running from the the low of $26.03 to high of $35.84, results did not vary greatly, but were only available for three program implementations. -
Graph 6: Dollars to pages. This bubble graph depicts how much content gets produced based on budget and participation rates. The size of the bubble, and corresponding number they are labeled with, show how many total printed pages were produced through three program implementations for which data were reported. It shows that the amount of content created varies, and the two points with the highest productivity per person each week also high higher budgets. It is not clear if budget makes a difference in how much content is produced; more data are needed.
- For the seven reported Wikipedia Education Program implementations, almost 3,000 different Wikimedia pages were created or improved.
The majority of program implementations (86%) reported how many pages were created or improved on wiki during the program. The total was 2,629 wiki pages created or improved. Three of the reported implementations (43%) were able to report the exact number of characters added by their program participants, allowing us to learn that across the 1,927 different Wikimedia pages created or improved for those three implementations, 16,820 pages worth of text were added.
In addition, we received data about how many photo/media files were uploaded to commons by program participants for use in Wikimedia pages for six of the program implementations (86%). Program leaders reported a total of 2,111 photos/media that were added, with each of the six implementations reporting from 15 to 1,139 photo/media files added with an average of 252.[23]
- Out of the 2,629 pages that were created or improved during six reported implementations, 1% had been rated as Good Articles.
We also wanted to learn what type of article quality was being created, and if contributions had been ranked as Good or Featured Articles. Only 35 (1%) of the reported 2,629 created or improved wiki pages were reported as Good Articles (see Graph 7) and none had become Featured Articles.
Recruitment and retention of new editors
[edit]- In total, 3,334 new editors participated in the reported Wikipedia Education Program implementations.
We also wanted to learn about the retention of active editors after they participated in the Wikipedia Education Program. While this wasn't selected as a priority goal in the pilot data collections survey, it was cited by some community members as being of interest, outside of this reporting process.
We asked program leaders to report retention rates for their participants three and six months after the event ended.[24] Active editors are defined Wikipedia editors who make an average of five or more edits per month.
We received data from:
- Six implementations (86%) about the number of active editors at 3-month follow-up, because those six programs had met the three-month mark for follow-up (Total 2,960 new editors tracked)[25].
- Five implementations (71%) about the number of active editors at 6-month follow-up, because those five programs had met the six-month mark for follow-up (Total 2,935 new editors tracked)[26].
- 1.2% of total new editor participants from reported Wikipedia Education Program implementations were "active"[27] three months after their course ended, and 1.1% were "active" six months after their course ended.
- 3-month active editor rate for the 2,960 new editors tracked ranged from 0% to 60% with an average of 2% retention (see Graph 8).[28]
- Out of the total 2,960 new editors recruited for the reported Wikipedia Education Program implementations, 1.2% (36) were active editors at the 3-month follow-up point (see Graph 9).
- 6-month active editor rate for the 2,935 new editors tracked ranged from 0% to 15%[29] with an average of 0% (see Graph 8).[30]
- Out of the total 2,935 new editors recruited for the reported Wikipedia Education Program implementations, 1.1% (33) were active editors at the 6-month follow-up point (see Graph 9).
- Out of the total 2,935 new editors recruited for the reported Wikipedia Education Program implementations, 1.1% (33) were active editors at the 6-month follow-up point (see Graph 9).
-
Graph 8: Retention box plot. This box plot depicts the full distribution of retention rates for each report. Reports were highly varied and ranged from 0% to 60% retention of active editors at 3-month follow-up and from 0% to 15% active editor retention at 6-month follow-up. However, the 50% of reported values occurring around the median of 1.7% ranged from 0% to 11.8% for 3-month, and from 0.9% to 6.7%, around the median of 0.04%, for 6-month retention, a much closer approximations of the most typical range for these projects. -
Graph 9: Recruitment and retention. The bars on this graph summarize the total number of education program participants who created new user accounts as part of the program and the 3- and 6-month retention rates for those new users. For all programming reported, the total number of new user participants was 2960, of which, 36 were active editors (i.e. 5+ edits per month) at 3-months, and 33 were active editors 6-months after the program.
- The Wikipedia Education Program can access new contributing editors through small, volunteer-run programs with little to no budget to large-scale programs with significant investments of resources (staff and volunteer time and money).
For many program leaders reporting on the Wikipedia Education Program, a priority goal was not retention of editors, but retention and growth of educational partnerships, whether between instructors and/or institutions. The programs are currently designed to maintain a steady flow of new students to continually contribute and produce content, rather than to convert each student into a long-term retained editor. Each program implementation had its own instructor and classroom count, often varying due to the length of the program and how new or old the program was in implementation. The number of instructor classrooms ranged from 1 to 112 for each of the implementations and was higher for on-going programs and programs established as on-going. For the on-going programs the number of instructors ranged from 12 to 112 while the newest pilot program implementation engaged seven instructors. The more instructor classrooms involved the bigger the pipeline of editors, still small single classroom implementations also engage new editors in contributing to Wikimedia through the Education Program.
Replication and shared learning
[edit]- The majority of Wikipedia Education Program program leaders identify as experienced, and reported producing resources, materials, and how-to's to aid others in implementing their own versions of the program.
We asked program leaders to share with us the types of shared learning resources were produced related to their program that could help others in similar efforts. Of the six program leaders reporting, 100% felt they were experienced and could help others conduct their own program as well as had produced blogs or other online information available for others to learn more about the projects. A smaller majority of program leaders reported that they developed brochures and/or printed materials (57%) and guides or instructions on how to contribute to Wikipedia for event participants (57%) (see Graph 10).
Next steps
[edit]- Next steps in brief
- Increased tracking and reporting overall
- Increased tracking of detailed budgets and staff and volunteer input hours by program leaders.
- Valuation of volunteer hours input and estimation of comparative costs.
- Use of better tracking systems for instructors, classrooms, and participant cohorts to examine the associated recruitment and retention.
- Increased ability to collect, and link to existing, demographic and experiential survey data.
- Increased ability to track pages created or improved and indicators of quality.
- Next steps in detail
As with all of the programs reviewed in this report, it is key that efforts are made toward properly tracking and valuing programming inputs in terms of budgets, resources, and hours invested in activities related to planning and implementing the program as well as comparative costs for content production. In addition, it will be important to institute better user tagging mechanisms to track particular participant cohort and retained members so we can investigate recruitment and retention and the possible intersections between cohorts and other programing streams. Program leaders focused on recruiting and retaining instructors may want to devise some methods to gather and monitor instructor experiences in the context of their length of participation as well as targets for the retention and growth in the number of instructors and classrooms. Lastly, further investigation of expectations and efforts directed toward the many other goal priorities is needed, including the goals related to improving accuracy, diversity of participants and content, as well as respect for Wikimedia projects. It will be important to identify possible quantitative or qualitative measures that relate to those important goals.
Summary, suggestions, and open questions
[edit]How does the program deliver against its own goals?
[edit]Survey respondents highlighted five priority goals for Wikipedia Education Program implementations:
- increasing accuracy and/or detail of contributions
- increasing contributions
- increasing diversity of information coverage
- increasing diversity of participants
- increasing respect for Wikimedia projects
We'll take each of these individually.
- Increasing accuracy
The goal of increasing accuracy gets at article quality. For some program implementations, the number of Good Articles and Featured Articles arising from student editor work is a good measure. [31]
To our knowledge, only one program leader has done any large-scale quality research into student work: the program in the United States and Canada. The first study reviewed 140 student articles and the second study reviewed 124 student articles. Both studies found student editors significantly improved article quality.
Determining better strategies for measuring quality improvement is a key area for development for program leaders focused on that goal.
- Increasing content
Data from this report shows the Wikipedia Education Program increases content. With an average of 120 printed pages of content each week, the Wikipedia Education Program makes many contributions to the growth of articles on Wikipedia.
- Increasing diversity of coverage
While there are no systematic measures in place, the Wikipedia Education Program seems to address this goal. It is possible to fill gaps in content using student editors. It is fairly straightforward for program leaders to identify content gaps on their language Wikipedias and can thus target specific course subjects (like humanities or arts) for participation in a Wikipedia Education Program to fill those gaps. In targeting topic-related course and asking students to edit on course-related topics, the education program can leverage the knowledge content of each classroom to fill such gaps. It is important for program leaders to establish tracking mechanisms to evaluate strategies and outcomes for this goal in the future.
- Increasing diversity of contributors
While we did not systematically collect data on diversity of contributors, in many countries in the world, women outnumber men in higher education settings. In a free-form "additional information" question, one program reported 61% female participation and another program reported 87% female participation. Wikipedia Education Program implementations with this goal should be better about collecting data on gender diversity among participants.
- Increasing respect for Wikimedia projects
Many program leaders target increasing respect for Wikimedia projects, such as affecting student and teacher attitudes about it as a reference source. We are unaware of any education program that has developed a way to measure this, although we know some work is being done to determine ways to measure institutional views of Wikipedia.
How does the cost of the program compare to its outcomes?
[edit]For the three program implementations for which costs and content production were reported, the costs per page of text added ranged from $26 to $36 USD per page, the higher costs were associated with newer program implementations. Although we did not probe into the variability of implementation effort for a new verses second or third rounds of education program implementations, it is important to note that start-up costs for staff-run programs can be high. For example, a case study for the Wikipedia Education Program Egypt, terms 1 and 2, the cost of the program was high in its first term; indeed, it would have been more cost effective to hire professional translators to add translated content to the Arabic Wikipedia. Once the program was better established, in its second term, cost per character decreased by 50%.
Without a better understanding of the value of time inputs to the content produced by more program implementations it is hard to say how the costs compare to the outcomes overall. It will be important to for Education Program leaders to increase their tracking and reporting of the input time and resources and the content produced by their program participants in order to make that determination.
How easily can the program be replicated?
[edit]The Wikipedia Education Program has extensive documentation, and currently, replication and adaptation is one of the Wikipedia Education Program's potential strengths. All of the reporting program leaders indicated they were experienced and felt comfortable mentoring new program leaders, and several program leaders have actually done mentorship calls with other program implementers. Additionally, the Wikipedia Education Program has many brochures and guides available which present best practice information for both instructors and students, as well as online trainings for program participants and new program leaders. However, these guides and learning pathways have not always existed (thus the current landscape of apples and oranges in terms of program designs). These guides are based on the large-scale, on-going program model established by the Wikimedia Foundation staff.
The Wikipedia Education Program is unique in that it has had a team at the WMF working with program leaders and educational partners to identify and establish a program model and resources for replication and adaptation (see below). It is worth recognizing that this may be only one of many potentially successful education program models. Future implementations should be evaluated both against their own self-established goals and program designs, as well as against this well-resourced model of implementation, in order to draw conclusions and comparisons.
- Brochures
The Wikipedia Education Program has created four brochures for instructors:
- Instructor Basics: How to use Wikipedia as a teaching tool
- Case Studies: How professors are teaching with Wikipedia
- The Syllabus: A 12-week assignment to write a Wikipedia article
- The Essentials: What to do before the term starts
An additional three brochures are available for students:
Although distribution of these resources is unknown, several of these titles have already been localized and translated into other languages. Source files are available for all brochures in both InDesign and Scribus.
- Online trainings
New program leaders can create a pilot program plan for starting an education program through an online training on Outreach wiki. The online training contains best practices from several program leaders' experiences and helps new program leaders develop and execute their program plans for a Wikipedia Education Program implementation.
Three online trainings exist for program participants as well:
- Training for students: a four-part training intended for students doing assignments on Wikipedia, with more detailed introductions to core Wikipedia policies, editing basics, and more specific editing advice for students.
- Training for educators: a four-part training for professors and other educators who want to run Wikipedia assignments for class, with introductions to core Wikipedia policies, editing basics, and an overview of best practices for designing and implementing Wikipedia assignments.
- Training for Wikipedia Ambassadors: a four-part training for Wikipedia Campus and Online Ambassadors, with introductions to core policies and editing basics for those new editing, and an overview of best practices for Wikipedia assignments.
These three trainings are available for porting to other language Wikipedias on Meta.
All of these materials contribute toward possible replication and growth of the Wikipedia Education Program globally and demonstrate the ability of program leaders to share in learning.
Appendix
[edit]Summative Data Table:Wikipedia Education Program (Raw Data)
[edit]| Percent Reporting | Low | High | Mean | Median | Mode | SD | |
|---|---|---|---|---|---|---|---|
| Non-Zero Budgets | 86% | $250.00 | $264,122.00 | $46,806.24 | $7,775.00 | none | $114,651.41 |
| Staff Hours[32] | 71% | 0.00 | 4000.00 | 1548.80 | 0.00 | 0.00 | 2122.71 |
| Volunteer Hours | 100% | 0.00 | 14000.00 | 3522.86 | 150.00 | 150.00 | 5905.72 |
| Total Hours | 100% | 0.00 | 18000.00 | 4629.14 | 150.00 | 150.00 | 7778.53 |
| Donated Meeting Space | 0% | Not Applicable - Frequency of selection only | |||||
| Donated Materials/ Equipment | 29% | Not Applicable - Frequency of selection only | |||||
| Donated Food | 14% | Not Applicable - Frequency of selection only | |||||
| Donated Prizes/Give-aways | 43% | Not Applicable - Frequency of selection only | |||||
| Participants | 100% | 25 | 2372 | 476 | 71 | 25 | 852 |
| Dollars to Participants | 86% | $0.83 | $370.15 | $136.84 | $66.67 | none | $164.46 |
| Input Hours to Participants | 100% | 0.00 | 29.37 | 6.83 | 3.00 | none | 10.33 |
| Bytes Added | 43% | 627762 | 18761380 | 11536981 | 15221802 | none | 9611995 |
| Dollars to Text Pages (by Byte count) | 43% | $26.03 | $35.84 | $29.86 | $27.70 | none | $5.25 |
| Input Hours to Text Pages (by Byte count) | 29%[33] | 1.77 | 2.20 | 1.99 | 1.99 | none | 0.30 |
| Photos Added | 86% | 15 | 1139 | 352 | 252 | none | 424 |
| Dollars to Photos | Not applicable - Not a key outcome of WEP | ||||||
| Input Hours to Photos | Not applicable - Not a key outcome of WEP | ||||||
| Pages Created or Improved | 86% | 38 | 1360[34] | 438 | 284 | none | 507 |
| Dollars to Pages Created/Improved | 71% | $0.40 | $631.87 | $174.81 | $100.67 | none | $261.25 |
| Input Hours to Pages Created/Improved | 86% | 0.00 | 43.06 | 10.22 | 3.85 | 0.00 | 16.49 |
| UNIQUE Photos Used[35] | 43% | 7 | 75 | 44 | 50 | none | 34 |
| Dollars to Photos USED (Non-duplicated count) | 43% | $3.33 | $2,142.86 | $719.06 | $11.00 | $3.33 | $1,233.05 |
| Input Hours to Photos USED (non-duplicated count) | 43% | 0.00 | 4.00 | 2.33 | 3.00 | none | 2.08 |
| Good Article Count | 43% | 4 | 20 | 9 | 8 | none | 0 |
| Featured Artice Count | 29% | 0 | 0 | 0 | 0 | 0 | 0 |
| Quality Image Count | Not Applicable | ||||||
| Valued Image Count | Not Applicable | ||||||
| Featured Picture Count | Not Applicable | ||||||
| 3 month Retention[36] | 86% | 0.00 | 0.60 | 0.12 | 0.02 | 0.00 | 0.24 |
| 6 month Retention[37] | 86% | 0.00 | 0.15 | 0.03 | 0.00 | 0.00 | 0.06 |
| Percent Experienced Program Leader | 100% | Not Applicable - Frequency of selection only | |||||
| Percent Developed Brochures and Printed Materials | 57% | Not Applicable - Frequency of selection only | |||||
| Percent Blogs or Online Sharing | 100% | Not Applicable - Frequency of selection only | |||||
| Percent with Program Guide or Instructions | 57% | Not Applicable - Frequency of selection only | |||||
Bubble Graph Data
[edit]Data for Graph x. Budget, Participation, and Input Hours.
| Report ID | Average Weekly Hours Input per Participant | Average Weekly Dollars per Participant | Bubble Size: Number of Participants |
|---|---|---|---|
| 9 | 0.18 | $5.50 | 50 |
| 30 | 0.75 | $2.75 | 25 |
| 35 | 0.15 | $2.14 | 2372 |
| 44 | 0.51 | $8.33 | 48 |
| 53 | 0.01 | $0.01 | 300 |
| 57 | 0.42 | $2.11 | 71 |
| 67 | 0.56 | $7.12 | 468 |
Data for Graph x. Budget, Participation, and Page of Text Added
| Report ID | Average Text Pages (by Bytes) per Participant (per week) | Average Weekly Dollars per Participant | Bubble Size:Number of Participants |
|---|---|---|---|
| 35 | 0.08 | $2.14 | 2372 |
| 44 | 0.23 | $8.33 | 48 |
| 67 | 0.26 | $7.12 | 468 |
More Data
[edit]Program Inputsː
| Report ID | Program Length (weeks) | Budget | Staff Hours | Volunteer Hours | Total Hours | Donated Space | Donated Equipment | Donated Food | Donated Prizes |
|---|---|---|---|---|---|---|---|---|---|
| 9 | 17 | 150.00 | 150 | ||||||
| 30 | 8 | $550.00 | 0 | 150.00 | 150 | Yes | Yes | Yes | |
| 35 | 52 | $264,122.00 | 4000 | 14000.00 | 18000 | ||||
| 44 | 37.5 | $15,000.00 | 0 | 0.00 | 0 | ||||
| 53 | 86 | $250.00 | 0 | 300.00 | 300 | Yes | Yes | ||
| 57 | 2 | $300.00 | 60.00 | 60 | Yes | ||||
| 67 | 52 | $173,230.00 | 3744 | 10000.00 | 13744 |
Program Outputsː
| Report ID | Number of Participants/
New Accounts Created |
Number of Classrooms | Number Bytes Added | Number Text Pages (by Bytes Added count) | Number Photos/media Added |
|---|---|---|---|---|---|
| 9 | 50 | 1 | 44 | ||
| 30 | 25 | 1 | 100 | ||
| 35 | 2372 | 112 | 15221802 | 10147.87 | 403 |
| 44 | 48 | 1 | 627762 | 418.51 | 15 |
| 53 | 300 | 12 | |||
| 57 | 71 | 3 | 410 | ||
| 67 | 468 | 61 | 18761380 | 6253.8 | 1139 |
Program OutcomesːQuality Improvement and Active Editor Recruitment and Retention
| Report ID | Number Unique Photos Used | Number Pages Created or Improved | Number Good Articles | Number Featured Articles | 3 month Active Editor Rate | 6 month Active Editor Rate |
|---|---|---|---|---|---|---|
| 9 | 38 | |||||
| 30 | 50 | 40 | 20 | 60% | n/a | |
| 35 | 418[38] | 11 | 0 | 0% | 0% | |
| 44 | 7 | 149 | 0 | 0 | 0% | 0% |
| 53 | 75 | 624 | 4 | 3% | 3% | |
| 57 | 0% | 0% | ||||
| 67 | 1360[39] | 4% | 4% |
Notes
[edit]- ↑ Making five or more edits per month
- ↑ One program reported on all their programming back to October 2011, including an additional ten months of participant data aggregated in their reporting. This went beyond the requested reporting window for Education programming from Fall 2012 to Summer 2013.
- ↑ Mean= 9, SD= 4
- ↑ Averages reported refer to the median response. Mean= $46,806.24, SD= $114,651.41.
- ↑ Mean= $1,329.61, SD= $1,794.90
- ↑ Mean= $4.00, SD= $2.66
- ↑ 'Mean= 87.1, SD= 99.1
- ↑ Mean= 74.5, SD= 2.0
- ↑ Mean= 99.4, SD= 126.2
- ↑ Mean= 22.2 minutes, SD= 13.8
- ↑ Mean= 476, SD= 852
- ↑ Mean= 36.4, SD= 29.8
- ↑ Mean = $136.84, SD = $164.46
- ↑ Mean = 6.8, SD = 10.3
- ↑ Mean = $3.74, SD = $2.78
- ↑ Mean = 22 minutes, SD = 13.8
- ↑ Note: Although "content production" is a direct product of the program event itself and technically a program output rather than outcome most of the program leaders who participated in the logic modeling session felt this direct product was the target outcome for their programming. To honor this community perspective, we include it as an outcome along with quality improvement and retention of "active " editors
- ↑ Mean= 108.86, SD= 65.62
- ↑ Mean= .19, SD= 0.07
- ↑ (Mean= 9, SD= 7)
- ↑ (Mean= 9, SD= 7)
- ↑ Mean= $29.86, SD= $5.25
- ↑ Mean= 1352, SD= 424
- ↑ Unlike in other programs reports of "active" editor retention, retention here was measured for activity in the third month and sixth month after course exit. In one case, the active editor count at six months was higher than at three months.
- ↑ usernames were only tracked for 144 out of 468 program participants in one qualified case
- ↑ usernames were only tracked for 144 out of 468 program participants in one qualified case
- ↑ Making five or more edits per month
- ↑ Mean= 12%, SD= 24%
- ↑ While editor retention is not a priority goal for many education program leaders, one program stood out with 15% new editors which were active six months after the program ended. Why? For most of the Wikipedia Education Program implementations reported, the proportion of new editors which were "active" at six months was much lower (0% for three program implementations and 3% for the other which had reached the follow-up point). However, for one specific program, the active editor rate was high; out of 144 total participants, 15% were active editors six months after the program ended. Programming context or design may play a role in this. This program was also shorter than the typical term-length program. It may be of interest to research context conditions (political and social climate, program length and timing, etc.) to see why this specific program had such high rates of active editors. There was a lot of variability in the the length of the Education Program implementations reported. Currently, this highest active editor rate associated with a much shorter length term set the 3-month follow-up point after class completion at five months following account creation while term-long engagements have a 3-month retention after class completion at 7 months following account creation. We need to standardize retention windows, meaning, we need to decide if we want to measure 3- or 6- month follow-up from the date participants create their accounts or from the end of the program or semester (as done here), and whether there are other intervals or types of follow-up of more important, or , the latter which was done in this report. Having consistency is important, so we can draw solid conclusions and comparisons across different education program implementations. Having dialogue with program leaders will be especially important as we work to develop these evaluation strategies.
- ↑ Mean= 3%, SD= 6%
- ↑ For other program implementations, it is not. For larger Wikipedias, Education Program leaders have found that backlogs in the GA/FA review process mean that student editors don't get feedback on their GA/FA nominations until weeks after the completion of the academic term, and reviewers are understandably frustrated when students no longer respond to their suggestions for improvements. Thus, many program leaders operating on large language Wikipedias actually discourage student participation in the GA/FA process. Those program leaders that focus on translations (including one program referenced in this report) use GA and FA articles as the source text for translation; while the translation may not go through the GA/FA process on the destination language Wikipedia, it had gone through such a review on the source language Wikipedia, leading us to conclude the accuracy of information presented is increased.
- ↑ Only 2 reported non-zero staff hours.
- ↑ Only 2 reported both bytes and hours.
- ↑ Note this number is only pages created; due to limitations in Wikimetrics, the number do not include pages improved. If it were to, the number would be much higher.
- ↑ Note: Photos Used reference is to unique, non-duplicated, count of photos used as of Nov. 4, 2013
- ↑ refers to new contributor retention only
- ↑ refers to new contributor retention only
- ↑ Note this number is only pages created; due to limitations in Wikimetrics, the number do not include pages improved. If it were to, the number would be much higher.
- ↑ Note this number is only pages created; due to limitations in Wikimetrics, the number do not include pages improved. If it were to, the number would be much higher.