Grants talk:IEG/Reimagining Wikipedia Mentorship

Where do lessons come from?

Will we be developing lessons as part of the work for this IEG or are we only building the structure in which lessons will fit and inviting mentors to add lessons. In other words, are we building course material or just the classrooms? --EpochFail (talk) 16:19, 23 March 2014 (UTC)

I think our focus should be on the structure by which mentors and learners can meet and successfully engage in a mentoring relationship. The building of course materials is what mentors can bring in; I think it will be good if we encourage existing approaches to teaching to be shared in an accessible space by mentors for mentors, but I don't think we have the resources to build a substantial amount of course material on our own as a function of this project. I JethroBT (talk) 19:17, 23 March 2014 (UTC)

Your focus seems wise, I JethroBT. Wikipedia is full of documentation, lesson plans, and tutorials for those who learn best via such means (and of course these materials can always be improved), but spaces designed for newbies to meet someone willing to provide personalized 1:1 mentorship are harder to come by. Siko (WMF) (talk) 23:26, 30 March 2014 (UTC)

Assessments

One topic that came up recently on IRC (#wpmentorship^connect) was that of assessments (exams, quizzes & tests). It sounds like some of us have strong feelings about whether or not assessments should be part of this new mentorship program. @Sbouterse (WMF) and Ocaasi:. --EpochFail (talk) 16:19, 23 March 2014 (UTC)

We need to give time to think about this one, so we likely won't have definite answers before the proposal is submitted. Here's my view: 'Tests' are standard for assessment, but also nobody wants to feel like they're back in school. I don't have a problem with tests being part of a resource center so mentors can use them if they choose but i doubt that it's best if they are standardized and required.

I believe at least a decent proportion of learners and mentors will prefer for qualitative or experiential 'assessment' where possible. If someone is creating a new article, then the 'test' could be the article they create. For images it could be creating a sandbox page with a gallery and various size or caption formats. We're teaching people to edit, not just think on policy questions; so tests fit in to that at points, but i doubt at all points.

At least, I'd encourage the resources that collect to include experiential 'assessments' options, demonstrating understanding by doing. What better way to learn about deletions than to read the policy and then participate in a few AfDs and have your mentor check out your work?

In any case, there are lots of different skills to learn and lots of different forms of assessment, so we can explore and offer multiple paths within a well designed space--support and flexibility :)

P.S.I was a tutor and teacher, so i have my own biases here and am open to all options in our discussion.

Last point, part of it is just semantics and branding to me: 'Check your understanding', 'Skill Challenge', 'Teach the mentor', 'Put your knowledge to use', 'Act on what you learned'... all can incorporate asking questions to gauge understanding, while conveying a different tone. Ocaasi (talk) 20:06, 24 March 2014 (UTC)

Using Snuggle to send invites

I'm cool with updating en:WP:Snuggle to be able to send invitation to new users who look promising. I like this strategy for identifying new editors to invite, but it would make it difficult to run the controlled experiment that I have proposed in the project plan. For the experiment, invites need to be sent randomly so as to not introduce a bias. I propose that we use something like en:User:HostBot to send invites for the experiment, but that we plan to make use both HostBot and Snuggle to send invites outside of the experiment. --EpochFail (talk)

@EpochFail: I agree that using HostBot is a solid approach. That said, are there particular biases you are concerned about in additionally using Snuggle to send out experiment invites? Is it possible to introduce an algorithm with Snuggle that would randomly assign invitations on top of HostBot? I JethroBT (talk) 19:23, 23 March 2014 (UTC)

@I JethroBT: The best way I could do this is to create a version of Snuggle that only shows a subset of users. Otherwise, mentors using Snuggle might send invites to people in the control condition. This isn't impossible, but it might be substantially more work. --EpochFail (talk) 20:00, 24 March 2014 (UTC)

Step 3: Design a social technology for matching mentors/mentees

@Ocaasi and Gabrielm199:, can you guys work on fleshing out this section? It sounded like you have ideas and maybe even some useful code already developed. --EpochFail (talk) 16:21, 23 March 2014 (UTC)

Mentees--> Learners

Admittedly, this is a minor point, but I've decided to call the participants in this grant "learners" as opposed to "mentees." I know it is natural to use "mentee" because we are using "mentors" to describe the people who are teaching, but I feel like we should focus on what participants in the program are coming to do: To learn one or several skills that help them reach their editing goals. That said, if anyone feels like there are places in the proposal where it makes more sense to use mentor/mentee, please feel free to revert. I JethroBT (talk) 15:54, 24 March 2014 (UTC)

What about 'pupils'? Thanks, Matty.007 (talk) 20:15, 28 March 2014 (UTC)

I like the move to "learners" - it has a nice feel, I can actually see someone wanting to be called this (not so much mentee or pupil) :) Siko (WMF) (talk) 23:12, 30 March 2014 (UTC)

Stipends

Hi all - I note from your participants section that Steven Zhang may be (also?) serving in an advisory capacity, and that seems like an odd thing to pay a stipend for. So I wanted to confirm that I understand how you're thinking about this: Is the line item #4 for stipends actually focused on research/analysis stipends (ie, Steven & Gabriel will be paid for their analysis work), and presumably other inputs they provide to the project can be considered a gift of volunteerism? Cheers, Siko (WMF) (talk) 23:12, 30 March 2014 (UTC)

Hi Siko, I can answer part of this one. Essentially, any advising I do would be as a volunteer, and it would be my research/analysis work I'd receive the stipend for. That's my understanding of it anyways :) Steven Zhang (talk) 10:02, 31 March 2014 (UTC)

Hi Siko. Steven's role as a grantee is indeed specific to his research an analysis work which will be relevant during the initial data collection from current mentor programs and during data analysis for our pilot. Also, I've noted that budgets of many grants have specified roles/titles for individuals on their budgets. To clarify, we decided against this approach, because in an initial meeting we had, what seemed more relevant was time each member was able to give providing their skills or doing the tasks they were interested in, rather than trying to fit them in a specific job description. But yes, it would be fair to say that the line item #4 happens to encompass research/analysis roles in this proposal. I JethroBT (talk) 21:48, 31 March 2014 (UTC)

As an update, Steven has elected to serve this project in an advisory role rather than a grantee role. That said, I have kept his statement below under the "Participants" section to demonstrate the value that his background will bring to this project. I JethroBT (talk) 05:26, 11 April 2014 (UTC)

Eligibility confirmed, round 1 2014

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for round 1 2014 review. Please feel free to ask questions here on the talk page and make changes to your proposal as discussions continue during this community comments period.

The committee's formal review for round 1 2014 begins on 21 April 2014, and grants will be announced in May. See the schedule for more details.

Questions? Contact us.

--Siko (WMF) (talk) 20:07, 7 April 2014 (UTC)

budgeting

Hello! I wanted to ask the following questions:

• how many hours per your estimates will programmer's and designer's work take? rubin16 (talk) 06:39, 13 April 2014 (UTC)
• is there any schedule for the works planned? rubin16 (talk) 06:39, 13 April 2014 (UTC)

Thanks! rubin16 (talk) 06:39, 13 April 2014 (UTC)

Hi Rubin16, thanks for your questions. Let me take them on separately:

• We estimate the programmer will spend about 75 hours on their tasks and the graphic designer will take about 100 hours on their tasks. We have planned concrete responsibilities for each of these roles already. For instance some of the programmer's tasks include:

• developing the system by which editors will be matched to mentors
• creating a profile builder
• help integrate design elements with mediawiki templates

I anticipate that the graphic designer's work will include some back-and-forth with our team with regard to theming and the design elements themselves. Major elements include the logo for our space and the main landing page, but will also include page backgrounds, profile design, and page navigation, among other components. We also expect the designer and programmer will need to coordinate together to ensure the space is implemented well.

• We have a preliminary schedule right now and are already beginning on some of the tasks for the sake of preparation. For instance, this week we are:

• identifying potential mentors
• developing the survey intended for previous mentors and participants in the adopt-a-user programs
• generating theme ideas for the mentorship space.

Some of our initial priorities starting in June include:

• identifying potential candidates for our programmer and designers roles
• hiring those individuals
• conducting interviews and longitudinal analyses of past editor performance in the adopt-a-user programs;
• devising and organizing a list of discrete editing skills that learners might want to be taught.

In July, when we have selected our programmer and designer, we will be discussing our needs with them and negotiate a schedule with them in regard to image drafts / alpha versions of tools that will be used in the mentorship space. The physical building of the space is likely to go through August as we conduct testing and work out bugs. Ideally, by September, we can begin the pilot, recruit editors into mentorship, and conduct data collection for which we will do final analyses in December. This summary doesn't enumerate all of the tasks we will be doing, but I hope it gives you an idea of how the project will progress. Please let me know if there is anything I can clarify. I JethroBT (talk) 03:01, 14 April 2014 (UTC)

Everything is clear to me, thanks a lot for quick response rubin16 (talk) 06:45, 14 April 2014 (UTC)

English Wikipedia AGAIN

Frankly I'm getting sick of all these paid initiatives to develop yet another mentorship tool on the English Wikipedia. We already spent unthinkable masses of money on the Teahouse; in the meanwhile all the other mentorship in all languages and projects goes on run by volunteers, whose work is not less valuable just because they don't write in English and they don't submit ambitious grant requests. Please find one non-English Wikipedia in need of software/research/other professional help for their mentoring projects and put yourself at their service, or just quit this money-sucking. --Nemo 07:29, 15 April 2014 (UTC)

Hi Nemo. I'm going to address what feels like a broader accusation here, rather than anything specific to this project. I can see that you're upset, but it doesn't seem fair to take it out on the people here - if you're able to adjust your tone, that might help anyone reading you to take your suggestions to heart in a more productive fashion. I share some of your frustration that we're not doing as much on non-English projects as we'd like. My team is always on the lookout for non-English projects to support, but building these connections in ways that are actually useful to people so that we can offer the kind of support they need for their context takes time; for example, I'm going to Jordan next month to meet with the Arabic community to learn more about what other support is needed there, building on the community consultations we've been running in the ARWP village pump lately, and based on those conversations I hope we'll generate some more new projects in that community. This doesn't just happen naturally on its own, as you rightly point out. If you have contacts in another language who you know might need support, or any other proactive suggestions to offer about mentorship projects in other languages with volunteers who need support to pilot something new, I'm super happy to hear them and put our heads together to see how we can support them. I imagine this project team might also welcome suggestions and connections for considering how to expand systems that are proven to work in one community into others beyond English in the future. Yelling at the people who do show up to participate in the only language context they know doesn't seem very helpful though. Cheers, Siko (WMF) (talk) 18:21, 15 April 2014 (UTC)

Nemo, I'd like you to consider that because a great deal of care and effort was put into making the Teahouse successful on the English Wikipedia (which is now run by volunteers), similar Teahouse spaces have been opened on the Bengali Wikipedia, the the Arabic Wikipedia, and on the the Urdu Wikipedia. The great thing is, in this project, the actual space will be relatively easy to implement across non-English Wikipedia projects because the tools will not require a great deal of translation. To the extent that adopt-a-user models are used elsewhere and create the same kinds of limitations, I believe our approach offers a constructive alternative. I agree that mentorship efforts on other projects are just as valuable and important as the ones we are doing. All we have done is identified major shortcomings in how we mentor individuals on the English Wikipedia and proposed a solution-- one that requires a substantial expenditure of time and skills from a small team. If this pilot is successful, I'd be very interested in reaching out to other projects to see whether ideas or components of our project would be useful for their needs. I JethroBT (talk) 18:42, 15 April 2014 (UTC)

I object also. I just got a survey notice on my userpage about this project. Wikimedia community volunteer time is valuable and I feel burdened because every researcher who sends out a survey is completely oblivious to the time burden this puts on the community members who complete them and the fact they they are asking the same questions that every other researcher asks, while not having a plan to share their results with other researchers who want to ask the same questions. Could the leads of this project please tell the human subject research committee of all institutions with researchers have an affiliation that they have received a complaint about undue burdening of their research subjects, and then report back with the advice those boards give for lessening the burden on volunteer time? This is not fair to the community. Please cease the surveying pending community approval. This is spam. Thanks.

I really appreciate your research and intent of what you are doing but please work with others and the community and do not seek to unilaterally present this project as beneficial without mitigating harms. Blue Rasberry (talk) 20:12, 25 May 2014 (UTC)

Blue Rasberry , I think you've gotten the wrong impression. This is not an academic research project. It is an initiative started by a group of Wikipedians who have reached out to academic researchers (of the Wikipedian variety) for support. I appreciate your complaints, but it would be a shame to punish this Wikipedian initiative for mistakes made by others. Also, I'd appreciate references to previous surveys of mentors you have taken. --EpochFail (talk) 20:27, 25 May 2014 (UTC)

The project is doing human subject research. All academic researchers working in the United States and sometimes other countries know it is taboo to do human subject research without third-party oversight. If you say that none of the people involved in this are affiliated with a research institution and this has nothing to do with the advancement of anyone's career in the academic research sector then I apologize now for being mistaken, but this looks a lot like a project managed by people who know human subject research guidelines.

Please take the first step and list for me what new user research you considered when designing your survey, because as a project coordinator you are presumed to have more agency and expertise than the human subjects you are recruiting. I trust you will agree that your doing this would be orthodox and placing the burden of proof on the human subject is unorthox. After you do some of that then I will continue the conversation after doing research on my own. Thanks. Blue Rasberry (talk) 20:50, 25 May 2014 (UTC)

Also, thanks for replying to me and for your interest in doing research. Your work is very valuable and I appreciate it a lot. Excuse me for being so direct. Blue Rasberry (talk) 20:53, 25 May 2014 (UTC)

Hey Blurasberry, there's no requirement for 3rd party oversight generally. The world is different for medical and psychological research due to the potential harm. There's certainly no taboo for performing survey based research that isn't asking about illegal or taboo (heh) behavior. I know that the US gov requires accredited/funded universities to set up an IRB to review human subjects research. It's important to understand that research is possible without being associated with an academic institution. I'd venture to guess that the vast majority of (tech) user research is performed outside of academia -- at private institutions like IBM, Google, Yahoo and Facebook.

I'm seeing you turning a question back on me without answering it. I'll remind you that you complained about getting survey requests that asked the same questions over and over again. I'm merely asking you to either point to an example of where you were asked questions about your mentorship activities or to admit that your complains are more relevant to other researchers and other projects. I'm not only asking this to prove my original point. I'm also the primary reviewer for RCom's subject recruitment review process (intended for non-Wikipedians FYI) and I'd like to minimize such requests if they are -- in fact -- common. Right now, the primary survey type that we tend to push back on regardless of other issues is the common, "I want to survey Wikipedians about their motivations to edit."

If you're seriously interested in the user research that informed this project and the survey that you were asked to participate in, I'd like to point you to the four citations referenced in the body of the proposal. However, if you are asking which studies lead us towards this work, I'd like to direct you to my publications -- specifically my interviews of Wikipedian mentors and Teahouse hosts for this paper -- and Gabriel's recent work, "Mugar, G., Østerlund, C., Hassman, K. D., Crowston, K., & Jackson, C. B. Planet Hunters and Seafloor Explorers: Legitimate Peripheral Participation Through Practice Proxies in Online Citizen Science." pdf. --EpochFail (talk) 21:20, 25 May 2014 (UTC)

I just realized I forgot to also point to the Teahouse research. See Morgan, J. T., Bouterse, S., Walls, H., & Stierch, S. (2013, February). Tea and sympathy: crafting positive new user experiences on wikipedia. In Proceedings of the 2013 conference on Computer supported cooperative work (pp. 839-848). ACM. PDF --EpochFail (talk) 21:23, 25 May 2014 (UTC)

Here is a study which asks overlapping questions.

• Research:Finding a Collaborator

Thanks for explaining your position to me. I respect it and I hold a contrary position for which I currently see no path to reconciliation with yours. You obviously outrank me and other community members with your RCom and WMF position so what can I do if you have judged your research to be ethical. I make a standing request that you get third party ethical oversight and cease leveraging the good will and trust that the Wikimedia community has in Wikimedia projects and academia to persuade people to give you time to advance your personal interests. If you ever choose to publish your work and you are asked if you are in compliance with human subject guidelines for your research, please tell them that you have received a complaint about unethical practices in your work and that it remains unaddressed. Please also request all of your colleagues that I have said the same to them - I am directing this request to you in your capacity as a WMF researcher. In the meantime, my request is that you cease all human subject research pending you find a mentor to oversee you. If you get a professional overseer then please let that person or organization know that I wish to talk with them. Thanks. Blue Rasberry (talk) 14:52, 26 May 2014 (UTC)

Bluerasberry, it's important to me that we have a productive conversation here. I'm not arguing from my "position" in RCom (more a mop than a gavel if any power is involved at all) and I never brought up my WMF status (note I'm posting from my personal account). I'd also like to point out that I am a volunteer on this project and that I have stated before that there is no scholarly angle involved for me. Honestly, my most selfish angle in this project is my hope that en:WP:Snuggle will be adopted and made use of.

Now, if for just a moment, I could address your tone. The only way I can imagine justification for your aggressive stance is a complete misunderstanding of what's going on here. The "practice" you're calling "unethical" is that of other Wikipedians (to whom I am only an advisor) who did not see it relevant to pass through a 3rd party review before asking their peers for help constructing a new service for other Wikipedians. Let me say it again just to make sure that it is clear. I am an advisor and not actively involved. I am writing no research paper about this project. The project coordinators of this IEG are Wikipedians.

So, should the project coordinators submit their surveying plans for a review? I'll side with their judgement (I recommend you direct your arguments towards them) and abstain from participating in the review. But in my opinion, doing so would largely be pro forma, and at best, would waste the time of both the project coordinators and the reviewers involved. (ping User:DarTar, I'm sorry to get you involved, but we could use a comment from an RCom member who is uninvolved with this project). --EpochFail (talk) 15:47, 26 May 2014 (UTC)

Oh! and also the study you cited (Research:Finding a Collaborator) is run by non-Wikipedian, academic researchers. The study description doesn't mention either a survey or discussion of mentorship practices in the description of their interview-based user study, so I can't imagine that their work would be relevant to the mentoring space that's being proposed in this IEG. Even if they did, I am not able to find a publication of any kind. In other words, it looks like this reference is not relevant. --EpochFail (talk) 15:47, 26 May 2014 (UTC)

Bluerasberry, I realize that my being in a PhD program might lead you to believe that this is for academic research, but I tried to point out in the message that this survey is to inform the creation of a tool intended to support Wikipedia. If this were for academic research and I excluded this fact in the survey recruitment email, it would be dishonest on my part and I could get into trouble. What I am trying to say is that this specific survey is not for academic research and therefore does not require IRB approval if the data is not intended for publication. On another note, and to justify the purpose of this survey, what would upset me is if people were to create a tool without giving a voice to the people they are building the tool for! This survey is just a tool to have conversation, it is not for academic research purposes. In terms of community approval, what do you mean? There are a number of Wikipedians that are interested in the development of a tool to support mentorship. The survey is one way to support this development. Is there another type of approval that you would like us to pursue? Also, what do you mean when you say that this project needs a “mentor”? A mentor for what, exactly? A mentor to observe and oversee how we are measuring the outcomes of the project? We have many advisors who can help us with that if this is your concern. Please let me know what you mean and hopefully I can address your question more precisely. Gabrielm199 (talk) 17:00, 26 May 2014 (UTC)

Hi Lane. I have to agree that you're applying a rigor and scrutiny appropriate to clinical trial research rather than informal qualitative research of Wikipedia adopters. It's entirely typical for community members to survey editors for limited purposes that improve the community. I did so for The Wikipedia Adventure, and I have done so twice for The Wikipedia Library. These surveys help us work better, they are optional, they come with appropriate privacy and disclosure protections and they are generally entirely benign in their subject matter (e.g. What do you like about the existing adopt-a-user program and what would you like to see changed?). If you don't like surveys, don't do them. If you don't want to be asked to do them, perhaps we need to create a list to avoid 'survey spam'. Still, I generally think surveys are immensely valuable in helping us understand our community and I've yet to see a need for known, trusted community members to visit a Research Review Board in order to talk to other Wikipedians in an organized way. Best, Jake Ocaasi (talk) 17:03, 26 May 2014 (UTC)

@Bluerasberry: I'm in agreement with what others have said above, so I'm not going to repeat them. I'm afraid that you are correct in that we're not likely to be able to reconcile this if you insist that we obtain some kind of IRB approval or ethical oversight in this project and that we report your complaint to authorities who have no relevance in this project. It's simply not necessary— this is not academic work. I am sorry that our survey was an inconvenience to you. I think Jake has made some good suggestions that we can discuss further. I JethroBT (talk) 17:45, 26 May 2014 (UTC)

┌─────────────────────────────────┘
Thanks for your attention. Let me restate my ethical concern. The time of Wikimedia community members is a scarce resource which exists in the commons to be a benefit for all. Wikimedia projects seek to aid community members to use their time to engage in community projects, like interacting among each other and developing content. The reason why surveys require ethical oversight is because large numbers of researchers seek to tap into the common pool of community volunteer time, route them away from natural community interaction, and bring them into unnatural data collection projects. It will never be possible for all surveyors to be able to recruit all the volunteer survey takers which researchers desire; consequently, if research recruitment is allowed at all through Wikimedia community channels, then there must be regulation to decide which researchers get to advertise their surveys and which researchers do not. Assuming that not everyone gets to send a survey through the valuable Wikimedia community communication channels, there has to be discrimination about what surveys are allowed and which are not. I have extremely low standards but I expect more regulation than none at all. RCom has already proposed some rules and I like that. Among the standards which I want to see developed are a commitment to documenting the surveys that go to the community, making an attempt to reduce redundancy in asking questions which have already been answered, and developing guidelines for determining how many times any given demographic can be solicited within any given time period. Gabrielm199, when I talk about a mentor, I mean someone not affiliated with a survey but who will give third-party oversight of it in the model of an IRB. I have objections to this survey, and if I am representative of any demographic, then this survey may be stressful to others also. My objections to this survey include the following:

• This survey asks some common questions and I do not see documentation that an attempt was made to seek this information elsewhere
• I cannot find documentation that this survey was occurring, and I worry that it was indiscriminately sent out without regard to the collective time investment that might be put into it by all volunteers or the collective stress pushed on the community by being asked to take another survey
• I have concerns about the quality of the survey because it seems not to have been copyedited by someone with sufficient experience in making surveys. I feel like little time was spent in planning this survey, and I want to see surveys developed such that the respondents will give valid answers that can be used by other researchers. I personally was confused by the wording of this survey.

When Aaron said that he was the "primary reviewer for RCom's subject recruitment review process" my response was fear, because he has a conflict of interest and is using his position and rank as an alternative to helping mediate my complaint. My complaint was that this survey was asking common questions, and when he is both an adviser for this survey and the person most responsible for judging that the questions were not common, I feel shocked that he would not instinctively say "Since I am an adviser on this grant-seeking project, my career and position are tied to this project; for that reason I ought not be the one to provide ethical review of it." I become very afraid when the people who are involved in project management are also intimately involved in its ethical review. I am also afraid that Aaron is both WMF and the RCom person who decides what constitutes ethical use of human subject research time, because the interests of the WMF frequently conflict with Wikimedia community interests and I wish that a community representative could hold the place he describes. I would not want a conflict of interest to occur in which someone at the WMF wants to recruit human subjects associated with their work, and then Aaron feels unconscious pressure to give ethical approval to that researcher's work. No one should be the last word in ethical approval over human subject recruitment for work which their employer has stake.

Ocaasi, my concern is not about privacy and disclosure nor am I opposed to research on Wikimedia projects or the mentorship projects. My concern is that there are only a few thousand active Wikipedians and merely hundreds of Wikipedians who volunteer to give mentorship. This study seeks to divert the time of this community's most valuable members with a survey that I felt was not well designed or well reviewed, and after one study is done on this demographic, I will argue that the time of this small community is tapped and that further survey interventions on this community be disallowed for some time. Since not everyone can have access to this community, I want the group which does have access to it to be conscious that they are asking for time from the few and most valuable people in the Wikimedia movement and that researchers spare no expense, thought, or resources in giving that group an enjoyable survey experience and structuring the data collected in such a way that other researchers with similar questions can use the same data without wanting another survey.

I am not advocating for oversight of the sort that pharma research has, but I would like acknowledgement that not everyone can survey the group of Wikipedians which this survey is targeting and that there is no forum in place for regulating how much community time can be solicited and diverted with on-wiki advertising. There should be limits for targeting the same demographics repeatedly. I want transparency and compassion for the survey respondents, because to give anything other than this is to get data in the short term but to alienate the pool of research respondents in the long term. The risk of harm of spoiling the trust of research participants is why this and all surveys on Wikipedia need ethical oversight. Surveys are not bloodless interventions and they should be taken much more thoughtfully.

On-wiki is not the best way to sort this - Jake and Aaron I am emailing you. Jethro and Gabriel please email me if you want to have voice or video discussion about this. Gabriel, are you coming to WikiConference USA this weekend? If so meet me there. Blue Rasberry (talk) 11:39, 27 May 2014 (UTC)

Lane, I appreciate this turn in the conversation. However, you have made unwarranted claims against my professional ethics. At the cost of losing this progress, I fear that I can't simply let these claims stand. You state "[EpochFail] has a conflict of interest and is using his position and rank as an alternative to helping mediate my complaint" I had no such goal in mind and I find it absurd that you read my comments that way. The topic of this conversation was over whether 3rd party oversight was necessary for this survey. If the survey for this project were to be proposed to RCom, I'd recuse myself as I have in the past. You'll note that I have already asked an uninvolved member of RCom to become part of this conversation. Short of the formal review process of RCom, I think I should be expected to show up to participate in discussions around the projects for which I am involved. I think it is very important that we draw a line between "the review" and the discussion we have here -- especially when making claims about ethical violations.

"I wish that a community representative could hold the place he describes" I both agree and am offended. I'd love it if someone else would take up this particular mop, but few are interested, so I end up doing most of the work. Further, I am a community representative. I put RCom's subject recruitment review process in place years before I became a WMF staff member. My goals were to mitigate survey fatigue, but to still let good studies take place. My role is as a coordinator/ambassador -- helping researchers get the necessary docs together for others to be able to make judgements. The process is specifically designed so that many people (as many people as I can get to show up) are involved in reaching consensus about a study before it is approved. My !vote carries no more weight than anyone else's. This work is not part of my official work for the WMF, so I continue to do it as a volunteer.

Finally, everyone is welcome to participate in RCom's discussions or to apply to RCom for membership and accept a reviewer's burden (again, more coordination than anything). If you're seriously interested, contact User:DarTar or me about becoming a member. --EpochFail (talk) 13:55, 27 May 2014 (UTC)

Hello EpochFail. I have been stressed out and I think it gave me a bad attitude. While I am still a bit on edge and have concerns, I would like to apologize for calling you out and personally criticizing you. My explanation for doing that is my bad attitude which is my own problem and I will cool it. I apologize to everyone else for bringing a negative attitude into this forum, and for anything personal that I said to anyone else.

I really appreciate everything that RCom has done. I am pleased with all of RCom's progress and successes, and feel that research on Wikipedia could not possibly be better managed or handled than it already is, and that progress is coming at a faster and more praiseworthy rate than anyone should expect.

I am not sure that I want to participate in RCom overall but I would volunteer to give comment on studies which do human subject recruitment. If I were to do so, I would like to normalize my reviewing style with you and anyone else who reviews such things, so that we could consistently give the same sorts of comments on the same kind of research. I feel that if we talked only a bit then we are likely to have the same opinions about almost everything. To "mitigate survey fatigue, but to still let good studies take place" is what I want also.

I must be missing information about your role on RCom. On the face of things, I thought that you were both personally responsible for the ethical oversight of this study as participant in this project and had conflicting responsibly for the oversight of this study as an RCom reviewer. I trust that you agree that ethical review cannot come from within a research team, and must be from a third party, right? The absurdity you see is somewhere in here - I see two conflicting roles but you have information which I do not have, and my lack of information made me see a problem which must not exist. Obviously you know your role and I am misunderstanding and we can sort that out later.

Thanks - I consider all my objections resolved. They seem not to have been based on valid concerns anyway. Because we have a path going forward, I have gotten relief. I want success for this and all other research on Wikimedia projects. I will be in touch with you online in other channels.

If anyone needs more information from me to bring this to resolution on the study end then please speak up and I will say more to resolve this. This project has my full support. Blue Rasberry (talk) 14:48, 27 May 2014 (UTC)

Aggregated feedback from the committee for Reimagining Wikipedia Mentorship

Scoring criteria (see the rubric for background)	Score 1=weak alignment 10=strong alignment
(A) Impact potential Does it fit with Wikimedia's strategic priorities? Does it have potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	8.3
(B) Innovation and learning Does it take an Innovative approach to solving a key problem? Is the potential impact greater than the risks? Can we measure success?	8.5
(C) Ability to execute Can the scope be accomplished in 6 months? How realistic/efficient is the budget? Do the participants have the necessary skills/experience?	7.8
(D) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support? Does it support diversity?	8.2
Comments from the committee: Fits strategic goals. Mentoring is important and leads to better editors. The problem this project will be solving is one of the key problems that broadly exists in all Wikipedia versions. Although it only focus on English Wikipedia, it will be a good sample and create the impact beyond the original project. A challenge regarding sustainability will be whether the tooling will fit the needs of more experienced Wikipedians who can become mentors. The financial risk is high, but so is the potential impact. Measurable and innovative approach to the problem. Budget is reasonable, scope may take a bit longer than 6 months, but not completely unachievable. The grantees have the needed experience. Lots of community engagement, which we appreciate.

Thank you for submitting this proposal. The committee is now deliberating based on these scoring results, and WMF is proceeding with it's due-diligence. You are welcome to continue making updates to your proposal pages during this period. Funding decisions will be announced by the end of May. — ΛΧΣ²¹ 23:55, 12 May 2014 (UTC)

Round 1 2014 Decision

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $22,600

Comments regarding this decision:
We look forward to seeing this experiment in mentorship move forward!

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.

Community of practice

I wish you good luck with this! I'm looking forward to seeing how you improve our ways to bring people into our community of practice. What research on or best practices in androgogy are you drawing from? Sumana Harihareswara 13:16, 31 May 2014 (UTC)

• I have not thought of using literature on androgogy, rather I have been thinking about it more from a expertise sharing perspective or how to support and incentivize knowledge sharing within a community of practice. I am sure other members of the projects might have some other thoughts on this. That being said, I would be curious to hear your thoughts on how the androgogy literature might inform this work. Gabrielm199 (talk) 20:32, 31 May 2014 (UTC)

@Sumanah: Hi Sumanah. In adopt-a-user programs, we are interested in seeing what approaches that have been taken to teaching certain skills and concepts on Wikipedia, and will be dedicating time to collecting those methods so that we can construct a set of resources for mentors to use in this program. However, the scope of the project is less focused on studying and drawing conclusions about andragogy; we already know that we have excellent mentors on Wikipedia who employ different approaches to teaching. Their reputation and experience working with new editors who have different backgrounds and needs is sufficient for this project. The idea of researching and experimenting with the underlying methods of teaching is such a large task, that it would probably qualify for its own grant! However, like Gabe has said, please let us know if there is any research you think we should check out. Thanks, I JethroBT (talk) 01:11, 1 June 2014 (UTC)

External research collaboration idea re. invites

My research group at Carnegie Mellon University has been working with (User:EpochFail) and others for sometime to better understand how to help newcomers become integrated into Wikipedia (see [[1]], [[2]]). One facet of this is to figure out the best ways to encourage newcomers to seek the help that they will need to become successful contributors.

We've been considering the trade-off newcomers may feel seeking help from experienced Wikipedians. On the one hand, newcomers may feel experienced members are knowledgeable. On the other hand, they may feel these users are judgmental and have difficulty relating to inexperienced contributors. If so, they may feel more comfortable seeking help from other newcomers, who will be less knowledgeable, but perhaps more sympathetic. We are interested in running a controlled experiment this summer, ideally during the month of July when we have a few undergraduate research assistants who will be helping. Our ideas for an experiment are still in the early stages, but one possibility is to vary how to frame invitations to an existing mentorship program (like the Teahouse or the mentorship program you are designing). These framings would either emphasize the program as a way to connect to and get help from other newcomers or alternatively as a way to get expert advice from more experienced members. Each of these framings may have different advantages. Framing it as a way to get help from other newcomers may encourage newcomers to participate because they feel they can ask the most basic questions that they might be too embarrassed to ask more experienced members. Framing it as way to get help from experience members may encourage newcomers because it might give newcomers confidence that they will get good answers to their questions. Currently some invitations to get help now emphasize expert advice from experienced editors {{Wikipedia:Teahouse/AfC_Invitation}} while others emphasize peer support from others newcomers as well {{Wikipedia:Teahouse/Invitation}}.

This sort of experiment might help us develop the best ways to recruit newcomers to a mentorship program as well as help us understand who newcomers want to turn to for information and the way they frame requests when they think they are talking to peers or experts. It should be easy to implement by using a custom message invitation template.

Our goals seem to be closely aligned with your proposal. Perhaps we can discuss possible collaboration and/or integration of our experimental goals. For example, I could imagine that this initial invitation experiment might help you design your recruiting strategy when you launch your revamped program and/or figure out who newcomers want to turn to to learn skills.

YTausczik (talk) 15:40, 11 June 2014 (UTC)

@YTausczik: Thanks for bringing this to my attention! This line of investigation regarding newcomer invitations and how they are framed is definitely of interest to me. I agree that there may be differences in how new editors respond to "peer support" vs. "expert support" invites. This could be something we'd try out after our pilot. Right now, we are planning on recruiting mentors who are pretty clearly in the "expert support" camp; they've been editing for several years, they can teach a wide range of editing skills and therefore can broadly engage learners with different editing goals. It's important to us that learners do not have to wait for a mentor who can teach them what they want to learn, which is a major issue with the adopt-a-user program.

I understand that the manipulation you are describing would be very simple to implement. Certainly, if the pilot is successful, I'd like to have more peer mentors join us, where we can better investigate this question. That said, this is just my opinion, and I'd like others to weigh in here (@Soni, Gabrielm199, Ocaasi, and Steven Zhang:) and continue this discussion. I JethroBT (talk) 19:30, 12 June 2014 (UTC)

Do you know about when you are planning to run your pilot? Also, if we were to independently run this sort of study using another newcomer support program (e.g. Teahouse) would this interfere with your pilot? --YTausczik (talk) 18:27, 15 June 2014 (UTC)

Conducting research about mentorship programs on English Wikipedia

A couple of months ago there was a conversation about whether or not the intentions of this project were primarily for research purposes or for supporting newcomers. This uncertainty emerged around the deployment of a survey intended to gather valuable information about existing newcomer initiatives so as to inform the design of the project. While the conversation pointed to the fact that both the survey and the project as a whole were born out of a desire to help newcomers and not out of academic pursuits, there has been some discussion of late about how data on the outcomes of this project and the process of its development would benefit the larger community of open online collaborative community practitioners and researchers. As a team we have already developed research questions for the purpose of evaluating the project and it would be a shame if we did not take the extra step to, after making the findings openly available, publish the findings in the settings of conferences or journals. As such, I would like to suggest, in addition to building a valuable tool for the movement, we also consider the possibility of publishing the research we plan to conduct around how this project compares to other mentorship projects and share the findings in the form of publications and open data sets for the benefit of future mentorship programs in Wikipedia and other projects like it.

This research has already been approved by the IRB at my home institution, Syracuse University, and I will of course submit this research proposal to the RCOM review process. Before I apply for RCOM review I wanted to post about the potential for publishing research here given recent conversation about the identity of this project and address any concerns people may have. Gabrielm199 (talk) 12:15, 18 July 2014 (UTC)

• My concerns are resolved and based on all information I have, I encourage you to do what you think is best. Did anyone raise concerns other than me? Blue Rasberry (talk) 15:16, 18 July 2014 (UTC)

Renaming it to 'peers'

Hi I JethroBT, Soni, Gabrielm199. I'm excited to see this project starting off.

Could you please consider renaming things 'peers'?

• Being 'mentored' by one mentor is a lot less fun than having a few 'peers' and finding new 'peers' whom I can help. :-)
• I don't like the patronizing approach, but collaborating with someone on the same level would be nice for many newcomers.
• I tend to find it unfair that someone is called a 'reviewer' or a 'mentor' where the newcomer is the very source of knowledge and partly a more experienced side, too - just perhaps not wiki markup-wise.
• Such 'mentorship' approach perhaps also overcomplicates the learning process (the status of a mentor may be perceived as a result of long hard learning and the newcomer remains a newcomer).
• Also, newcomers should be encouraged to join the helping crowd -- there should be no entry barrier (while 'becoming a mentor' sounds like a very very distant task).

I believe this would make no difference to the essence of the program, but would make it sound more relaxed to everyone involved. --Gryllida 13:34, 1 September 2014 (UTC)

@Gryllida: Hey Gryllida, thanks for your thoughts. The terminology we've been going with so far has been "mentors" and "learners," based on how they are participating in the Co-op, but I really like the idea of considering all participants as peer editors. I agree that there's an unfortunate hierarchical component of mentoring, which can come off as patronizing. On the flip side, a new editor might want someone experienced who they trust knows what they are doing, and knowing that someone is a mentor makes that easy to recognize. That said, I think there are ways we can accommodate the "peer" name while while making it clear that we're selecting matches who are experienced in the appropriate area. Like the Teahouse, we also want to make the bar for teaching to be very informal rather than having to go through a set of prerequisites or fulfill some arbitrary edit count criteria. As for our general mentorship approach, we think there is a lot that a 1-on-1 approach to learning can offer, so there does need to be some way to distinguish between editors who want to mentor, and editors who are there to learn. Editors may want to do both, and we certainly want to encourage that. We have given a great deal of thought about the sustainability of the project, which will be very much dependent on making sure we encourage editors who use our space to learn can also teach others what they've learned. I JethroBT (talk) 05:18, 2 September 2014 (UTC)

I JethroBT: yeah, I'd start by renaming this page and rewriting some of its documentation. I can even do that for you, if you like.

Using the word 'peer' is healthy in long-term. And "making it clear that we're selecting matches who are experienced in the appropriate area" is not really hard -- everyone would be excited to participate as a peer and we'd get things going from there. Editor engagement 2fold: both helping and getting more helpers. :-)

(BTW, I would ideally have people gather around wikiprojects; I haven't used teahouse for this reason, although it is nevertheless a useful tool.) --Gryllida 05:02, 3 September 2014 (UTC)

That's very generous of you to offer to help! If you'd like to be bold and start to reframe some aspects of the grant proposal, please do, but our main hub is at en:WP:CO-OP, where I think our focus is right now in terms of informing folks about our project (i.e. this proposal page is a bit long and detailed for most). I've started to make some changes over there, but I'm welcome to your suggestions. I think I'd rather keep the grant name the same, in part because it would be confusing to the WMF if we renamed it well after we've been awarded the grant, but also because I think considerations around vocabulary and status are an important part of this reimagining process as well. I JethroBT (talk) 07:12, 3 September 2014 (UTC)

Scope

I hope it's being programmed in a way that makes supporting other languages and sister projects easy. Especially languages. :-)

--Gryllida 13:36, 1 September 2014 (UTC)

We're still looking to hire a dedicated programmer for the space, and if you know anyone who is interested, feel free to send them my way. Any suggestions about ensuring better transferability to other projects is welcome; if the space is successful during this pilot, I want to spend time making sure we can provide a toolkit for other Wikipedia projects if a mentorship space will help them. Right now, we have a programmer dedicated to putting together the matching gadget for the project. This gadget will basically be pulling information from inputs that editors provide when initially coming to participate in the space (whether they are learning or mentoring). Conceptually, the matching is just a matter of pulling information from these fields, so the translation component is not very intensive from that standpoint in the programming. I JethroBT (talk) 05:29, 2 September 2014 (UTC)

What programming languages is he required to know? I might suggest writing down the specs for the software you'd like to build. It's easier to send contributors here for a specific task. :-)

Please don't ping me, I have Echo turned off.

--Gryllida 05:03, 3 September 2014 (UTC)

Thanks for the note, sorry about that.

We'll need someone who has experience writing UIs and has experience writing in Javascript, HTML, CSS. Someone with bot experience on-wiki will be quite valuable. Unfortunately, if I had good specs ready, I'd be tossing it out there like hotcakes. We're still working on them based on the use cases we've developed so far. In a nutshell, the important elements include an interface that displays and allows selection of editing skills for editors to choose to learn/teach, a profile building system and page that retains user inputs to allow for matching, a bot that will contact peer editors to let them know they've been matched, and a tracking system so that editors can see what skills they have learned through the Co-op and what they are willing to teach. I JethroBT (talk) 07:37, 3 September 2014 (UTC)