Brazil Program/Education program/Goal

Goals

See First semester of the 2012 pilot

Second semester 2012:

Expansion: expand the number of classes to 12-15 with the learning points of the first phase of the pilot
Sustainability: answer the question on how sustainable is the program in the actual format for Brazil and Wikipedia in Portuguese context
Tutorials: create and improve tutorials in Portuguese for the program so that professors and students willing to adopt Wikipedia as a teaching tool can do by themselves
Impact: measure the program impact for bringing new contributors to the Wikipedia in Portuguese
Program ahead (2013): think of other formats an educational program in Brazil can improve the impact for educational purposes on (a) readers and (b) editors (new and experienced)
Documentation: improve in depth learnings about the program

WikiProjects

Rationale: We are going to measure the Medicine WikiProjet on the Portuguese Wikipedia after outreaching 60+ school hospitals and mailing lists of the Ministry of Health network.
Period of measurement: 22 April to 22 July (3 months) (Note: Starting date depends on some pending things and our partner, see here)
Bot for welcoming people: we will be using a bot for welcome to the WikiProject those who edit medicine article, see the repository of the code.

Logistical considerations

Tracking invitees

Email outreach

We need to make sure that we can identify which of the people who were emailed subsequently created a user account and (hopefully) participated in the project. We can ask people to send us their username once they've created an account, or to collect the usernames from their students, but realistically few people will do this. Tracking pageviews requires special preparation, and still doesn't tell us whether the person created an account or not. Another way would be to match the email addresses on the email list with the email addresses attached to newly-created user accounts. This may still miss some people (if they attached a different email address when they created their account), but at least we would be certain of the identity of those whose emails matched. We could also hopefully prevent some email mis-match by specifically asking people to use their work email address when registering in the body of the email itself. Or we could link from the email to a custom 'landing' page with instructions on how to create their account, and make the request there.

There may be more certain and systematic ways of doing this, too. Perhaps some members of the Education Program have tips, tricks or tools that could be of assistance.

New user outreach with WikiProjectBot

WikiProjectBot will need to store the user id of each user it invites, as well as the date at which they were invited. Other datapoints that could be captured at the time of invitation are useful but are less critical: what matters most is the user id and the invite date! The table below lists a set of datapoints about each user that it would be good to track from the beginning of the pilot.

data point	description	notes
user id	the id of the user	necessary!
invite date and time	the date and time that the invitation was sent.	necessary! This datapoint is important because a) you will need to know how many invites are sent out each day and b) when you analyze the editing behavior of the users, you will need to separate the edits they made before being invited from the ones they made afterwards.
user name	the name of the user	It is much easier to perform spot checks (to tell if the script is running correctly, or to check the contribs of an invited user) if you have their username in addition to their user id.
user editcount at invite date	how many edits the user had made at the time they were invited.	this is not necessary, but is much easier collect it from the `user_editcount` column in the `user` table (or through the API) at the time you send the invitation than to try to calculate it later.
first revision to WikiProject Medicina article	revision id of the edit the user made that caused them to be automatically invited	this is useful for spot checks: you want to be able to go back and look at what users were doing that caused them to receive an invitation. It will also be useful later for answering questions like What Medicine articles do new users like to edit? and Do new editors go back and edit the same Medicine articles after they join the project?
user registration datetime	when the user created an account	not necessary, but you'll need it eventually anyway and it's available through the API and through the database. Why not just capture this data now?
control group flag	whether or not the user was assigned to the control group and didn't get an invitation
subsequent edit flag	assigned after date of invitation. whether or not the user made at least one edit to Wikipedia after receiving the invitation	newcomers leave Wikipedia at a high rate. If a user had already stopped editing by the time they received the invitation, you don't want to include them in your analysis.
visited project flag	assigned after date of invitation. whether the user made at least one edit to the WikiProject Medicina pages	You need critical mass to keep a project going, so you will want to get an idea of your response rate early on. If the response rate is very low (<5%), you should consider inviting more people every day.

Encouraging them to participate

Only a fraction of the users you invite will ever visit the WikiProject, so you want them to feel at home. Here are few tips for engaging them. Some of these may seem obvious, but they are still often neglected! And you may only have one shot at 'hooking' these new users, so you want to get it right the first time.

Encourage users to introduce themselves on the talk page. Put a big link to the talk page on the front of the WikiProject (remember new users don't necessarily know what talk pages are about). And put a welcoming banner up at the top of the talk page, and/or include an edit notice when a user creates a new section that encourages them to say who they are and what they're interested in. You may want to have a separate forum for intruductions, like the Teahouse does, but that's not necessary if you can get people to introduce themselves on the talk page.
Encourage users to ask questions on the talk page. Mention in the invitation that they can go to the WikiProject with any questions they have. They should feel like this is their home base on Wikipedia, so even if the question isn't about Medicine they should be made to feel comfortable asking it.
Respond quickly, and be detailed and friendly. There should be editors watching the page daily and ready to answer questions (or just respond to users who drop by and introduce themselves, even if they don't ask a question).
Follow up with them on their user talk page. When they ask a question, leave a talkback notice. When they introduce themselves, give them a welcome message (or even a badge!). New users don't always understand that they need to go back and check the page for a response, or remember to do so.
Give them simple tasks to do, and offer support. Provide a list of simple tasks they can perform to begin with, and let them know that they can come back any time with questions or to have someone check their work and provide feedback.
Encourage them to join the project 'officially'. Let people know who the members of the project are by featuring the member list prominently on the project page (don't tuck it away out of sight). Encourage people to add themselves and make it easy for them to do so.

Ways of measuring impact

Quality

Ways to measure the impact of this initiative on article quality.

Number of articles created during study period in the Medicine category, compared to the moment before the WikiProjects restructuring.
Number of featured and good articles that WikiProject participants contributed to.
Number of articles evaluated by WikiProject participants during the study period: measure the number of articles that have Medicine rating templates added to their talk pages.
Article Feedback Ratings: have experts (medical professionals, professors, graduate students, etc.) rate a set of medical articles which using the Article Feedback Tool before the study starts. At the end of the study period, have this set of professionals rate the same articles. Compare their before/after ratings. (Discussion here)

Activity

Ways to measure how many people participated, and what they did during the study period.

Number of new editors who join WikiProject medicine during the study period. Should try to keep track of who joined because they were invited offline (for instance, a member of a university class) or online (for instance, if they received an automatic invitation).
Number of new editors who edited the WikiProject talk page, or who edited WikiProject medicine articles. You could break this down into different "buckets" if you wanted, such as:

made > 5 edits per month to the WikiProject pages
made > 5 edits per month to medicine articles
made > 100 edits per month to the WikiProject pages
made > 100 edits per month to medicine articles

number of Medicine articles edited by WikiProject members
number of topics being discussed on WikiProject talk page per month

Retention

Ways to measure how many people continued to be active on pt.wiki after the study period, and how active they were.

number of project members who are still active on Wikipedia (5 edits-per-month, 100 edits-per-month levels) one, three and six months after the study period ends.
survey project members and ask them what they liked, what they disliked, what was difficult, etc. about participating, and what would make them want to participate more.
number of project members who still make 5 or more edits to medicine articles, or still edit the Wikiproject page 5 or more times per month after the study period.

Number of active veteran Wikipedians: >= 5
Number of new contributors that become active: >= 15
Number of topics being discussed on the WikiProject talk page: >= 3 / month
Acomplish tasks in group: >= 3
Number of articles considerably improved by the group: >= 20 (until June)
Increase featured articles: >=2? (proposal)

Control group

To determine whether participating in the WikiProject had an impact on editors' behavior (such as their activity, the quality of their edits and their retention), you need to compare editors who participated with editors who did not participate, but could have. The best way to do this is to define a control group. This may not be possible or advisable for the users who are being recruited by email, but it is easy to do for editors who are being invited to participate by WikiProjectBot. Here's how to do it:

Every day, WikiprojectBot gathers a list of all the editors who met the criteria for invitation to the Wikiproject. So (in this example), that list contains 65 registered users who a) have fewer than 100 edits (newbies) and b) edited a medicine article within the past 24 hours. The bot then invites 50 of those editors, and records them as invited in its database. It also records the usernames and the date for the 15 editors that it didn't invite, and flags those records as control. Over time, the bot builds up a list of editors who were never invited to participate in the project--the control group.

At the end of the pilot, you filter out those editors in the control group who found their way to the project anyway by checking whether they edited the project page or talk page. For the rest of them, you can then count the number of edits they made, the number of weeks they were active, the amount of content they contributed, etc. By comparing these averages to the averages for the bot-invited editors who participated in the project, you can determine with some degree of certainty whether participating in the project had an effect on what the participants did on Wikipedia.

There are other considerations to keep in mind when filtering and comparing these groups, but that's the gist of it.

What success will look like

Under construction

If this pilot is successful, we will see evidence of that success in the following ways:

1. Impact on Portuguese Wikipedia

The amount and quality of content on the Portuguese Wikipedia increases

Measure of success #1: At least 70% of the articles proposed by professors will have students contributions
Measure of success #2: At least 75% of the content contributed by students is moved to the mainspace and "survives" (i.e., is not deleted from the Wikipedia namespace).

Number of active contributors during the pilot

Measure of success #1: At least 70% of the students or student cohorts supposed to contribute will engage on Wikipedia activities.
Measure of success #2: At least 50% of contributing students become active editors during the months of Wikipedia activities.

2. Structures for success

Various tools are in place to support the launch of the pilot!

Measure of success #1: The Brazil Pilot starts on-time (August 2012). Done
Measure of success #2: Training sessions are conducted by volunteers for the professors, campus ambassadors, and online ambassadors. Done
Measure of success #3: Surveys are created and distributed to professors and students at the beginning and end of the semester.
Measure of success #4: Feedback from training session is gathered via a post-training survey.
Measure of success #5: Clear documentation is created showing different options for assignment (e.g., editing existing articles, creating new articles, translating articles).
Measure of success #6: Clear documentation is created showing different options for Ambassador models, and pros/cons.
Measure of success #7: We will have numbers that clearly show which model for in-classroom activities is more successful (in terms of both amount and quality contributed).
Measure of success #8: We will have numbers and information to analyse how effective the program is in Brazil and which models work best.

3. Community impact and reaction

The Brazilian Wikimedia community and the Portuguese Wikipedia community are supportive of the education program!

Measure of success #1: Increase of at least 50% (?) in the number of wikipedians involved in the program both as campus and online ambassadors in comparison to the first semester
Measure of success #2: XXXXXXXXXx
Measure of success #3: XXXXXXX

The Brazilian Wikimedia community and the Portuguese Wikipedia community are happy/satisfied with their involvement in the pilot!

Measure of success #1: More than 75% of the community volunteers involved in the pilot indicate that it was worth their time. (Source: post-semester survey)
Measure of success #2: Members of the Portuguese Wikipedia indicate support for continuing to build an education program in Brazil (Source: post-semester survey)

4. Professor and student reaction

Professors and universities are receptive to the program!

Measure of success #1: We increase in more 100% the slots for professors - at least (10 classes).
Measure of success #2: 3 prominent universities participate in the program. Done

Both students and teachers feel more comfortable with editing Wikipedia!

Measure of success #1: 100% increase in the level of comfort. (Source: pre-semester survey vs post-semester survey)

TO BE CONTINUED

Students and professors are happy / satisfied with their involvement in the pilot

Measure of success #1: 75% of the participants indicate happiness with the outcome. (Note: We will conduct a survey after the semester ends to estimate this)
Measure of success #2: More than 50% of the students indicate that they would prefer a Wikipedia assignment type course to a traditional course in the future. (Source: post-semester survey) (DISCUSS)
Measure of success #3: More than 50% of the teachers indicate that they would use Wikipedia as a teaching tool again. (Source: post-semester survey)
Measure of success #4: More than 50% of the teachers indicate that they would recommend using Wikipedia as a teaching tool to their colleagues/peers. (Source: post-semester survey)

5. The road ahead

After the pilot, both community and staff members have a clear understanding what the road ahead for the Education program in Brazil

Measure of success #1: Bi-weekly updates on the program are shared with the PT:WP community; we publish at least 3 interim reviews with stories of success and lessons learned on the WMF blog.
Measure of success #2: An outcome / lessons learnt / the road ahead document will be completed by July 2012.
Measure of success #3: We will identify which in-classroom activities are most successful (in terms of both amount and quality contributed).

Background

See background.

Learning

See Brazil learning points.

Outreach