Learning and Evaluation/Archive/Connect/Questions
This is the parlor discussion page.
 | This page is kept for historical interest. Any policies mentioned may be obsolete. If you want to revive the topic, you can use the talk page or start a discussion on the community forum. |
Rationale for the privacy form
[edit]I can not understand the rationale for having participants in a course/event/edit-a-thon to fill a form to opt-in in Wikimetrics. In principle I could get any set of random user names from the history of any page, create a CSV file, upload it on Wikimetrics and analyse the users' activity. Without even meeting anybody. So why for events do we need an opt-in form? Again, in principle I could do this kind of analysis downloading the complete dump of a given edition of Wikipedia and write some tools to calculate the some things that Wikimetrics does. All that without asking anybody. Given this premise, with Wikimedia Italia we organized recently a course for librarians (see the project page) where every participant created a personal account. We then asked everybody to put their usernames in that page so that we could monitor their progress with Wikimetrics. We did not know about this opt-in form until today and so they have not signed it, even if they have been explicitly warned during the course. What can we do? Thank you. -- CristianCantoro (talk) 11:37, 10 December 2013 (UTC)
- Thank you Cristian for your comment and chance for clarification. You are correct that, in many cases, participant user names are a matter of public record, in that case it may not be necessary, but a matter of operating in a completely open manner in which participants actually approve of their being tracked and monitored via Wikimetrics which make it rather simple to do so. Importantly, Wikimetrics increases accessibility to metric data in a way that user names are transferred to Wikimedia Foundation (internationally to the United States) and to best cover everyones privacy interests and protections we need to allow them to opt in, as well as out, of this monitoring when we know we are collecting usernames explicitly for this purpose. The particular language and guidance for disclosing the who, what, where, and why that have been thoughtfully crafted by the WMF legal team to be sure privacy interests are protected and transparency is upheld. This is a shift in practice that we are working to assist program leaders in coordinating so that the future collection of usernames will best protect all parties interested. The usernames you already have public record of are still useable, we just recommend that you also allow people to opt out should they become aware of this usage of their usernames and wish to be excluded from monitoring (you may want to post the language and possibly instructions for "opting out" on the page they added their names to). I hope this helps, please let me know if I can answer any further. JAnstee (WMF) (talk) 21:58, 10 December 2013 (UTC)
- Hi User:CristianCantoro and JAnstee (WMF). I'm going to give a quick "TL:DR" about this:
- It's so we can be more transparent with the participants so they know what we're doing and in case they have concerns we can cover our butts with opt-in form documentation and terms of service (if they sign up on wiki).
- Perhaps it's different with your events, but, for me, sometimes people come to the event and don't sign up or even participate on the event page (but they edit at the event). Also, sometimes you'll have people sign up online but NOT attend/participate (online or at the event). So having an opt-in sign in sheet is an additional way to make sure everyone who attended is accounted for.
- And yes, like Jaime said - perhaps someone realizes that they signed up on the wiki event page, and they'd rather remove their name and opt out of being in the cohort. While sure, it's available and you're savvy enough you could dig and not tell them - transparency is the key. Some people - especially newbies - might not be aware of what happens after they sign their name on that event page, and perhaps they don't want to be a part of the research. Even if it's there in the history, it's important that we give people the option of being used in research or not. It's about being honest, transparency and respectful of participants.
- Hope this helps :) SarahStierch (talk) 22:30, 10 December 2013 (UTC)
- Also, User:Harej has started using the opt-in form at the Wikimedia DC events, which we share on the opt-in documentation page. He might even have some additional insight into why it's important for program leaders to use. Or he might not :) SarahStierch (talk) 22:31, 10 December 2013 (UTC)
- Sarah and Jaime covered most of it. Part of it is that the on-wiki list of users is not comprehensive, since some people create accounts on the spot and others attend without signing up ahead of time. The main benefit, however, is that it makes our work easier. I would rather have a form generate a list of usernames for me than have to scour Wikipedia to put together a list. This helps save time, which is great because we're all volunteers and want to do quantitative analysis with as little effort as possible ;-). It also allows me to conduct surveys of attendees; I'm especially interested in measuring what percentage of attendees come from the organizations we co-host events with, but you could also ask other questions. harej (talk) 00:48, 11 December 2013 (UTC)
- Also, User:Harej has started using the opt-in form at the Wikimedia DC events, which we share on the opt-in documentation page. He might even have some additional insight into why it's important for program leaders to use. Or he might not :) SarahStierch (talk) 22:31, 10 December 2013 (UTC)
- Hi User:CristianCantoro and JAnstee (WMF). I'm going to give a quick "TL:DR" about this:
- No such form is mentioned in the Wikimetrics documentation on mediawiki.org and if I try to login I get an error 500, could someone clarify what's required where to whom? What if I want to get some stats on WLM participants as hinted by Erik, or something like that? --Nemo 10:17, 11 December 2013 (UTC)
- Hi Nemo! Great to see you here. We're not responsible for maintaining the mediawiki documentation. I think that is a task for the analytics team. We actually do discuss it in our module, however, that was built and developed by the Program Evaluation and Design team, not by analytics. I'll ping them and let them know your concerns. I can't really comment about Wikimetrics and Wiki Loves Monuments, I'll defer to others to respond about that. SarahStierch (talk) 18:01, 11 December 2013 (UTC)
- As mentioned in my response above, usernames that already exist as public record are a grey area when it comes to this guidance for program leaders conducting in-person events and their collection of user names for use in tracking and monitoring contributions via Wikimetrics. Technically everything on wiki is public already, however, WikiMetrics introduces an ease at tracking and monitoring online activity and changes the game a bit. Importantly, once we are interfacing with community members in-person and collecting names of people who participate in those events we introduce direct data collection for which there may or may not have been a public record. There are some important precautions that we should then take to protect any participation information that may be sensitive. When we do this systematically, with intention to track and monitor, as we are guiding program leaders toward, we should disclose that we are doing so and give people the option to opt in or out. Also important, it is my understanding that this "opt in" is not a requirement of all users of Wikimetrics, but guidance for program leaders collecting usernames of participants of their in-person events. JAnstee (WMF) (talk) 21:01, 11 December 2013 (UTC)
- Ah, good to know, then I can ignore the Meta-Wiki page and refer to the official docs. Thanks, Nemo 22:39, 13 December 2013 (UTC)
- As mentioned in my response above, usernames that already exist as public record are a grey area when it comes to this guidance for program leaders conducting in-person events and their collection of user names for use in tracking and monitoring contributions via Wikimetrics. Technically everything on wiki is public already, however, WikiMetrics introduces an ease at tracking and monitoring online activity and changes the game a bit. Importantly, once we are interfacing with community members in-person and collecting names of people who participate in those events we introduce direct data collection for which there may or may not have been a public record. There are some important precautions that we should then take to protect any participation information that may be sensitive. When we do this systematically, with intention to track and monitor, as we are guiding program leaders toward, we should disclose that we are doing so and give people the option to opt in or out. Also important, it is my understanding that this "opt in" is not a requirement of all users of Wikimetrics, but guidance for program leaders collecting usernames of participants of their in-person events. JAnstee (WMF) (talk) 21:01, 11 December 2013 (UTC)
- Hi Nemo! Great to see you here. We're not responsible for maintaining the mediawiki documentation. I think that is a task for the analytics team. We actually do discuss it in our module, however, that was built and developed by the Program Evaluation and Design team, not by analytics. I'll ping them and let them know your concerns. I can't really comment about Wikimetrics and Wiki Loves Monuments, I'll defer to others to respond about that. SarahStierch (talk) 18:01, 11 December 2013 (UTC)
Hi JAnstee (WMF), SarahStierch, harej. Thanks for your answers (I am sorry for my late answer). I think that we can evaluate from case to case what to do, in general we never collect data such to be able to relate a given user name to a given person (i.e. a name and surname) and, in general, we have (so far) always used only publicly available data. I understand that in the case in which we would collate public data with non-public data (e.g. name, surname, age, profession) we would need some kind of opt-in and privacy statement. Wikimedia Italia already has its own privacy policy following the Italian law. Also, some of our members noted that such a statement could scary (unjustifiably) some people if we present it when just using public data and also add a distorted impression of "bureaucracy" in the Wikimedia projects. So we should use it only when needed (i.e. when requesting data which are not already public). -- CristianCantoro (talk) 14:32, 16 December 2013 (UTC)
- Hi User:CristianCantoro! I do want to clarify, just so we are all on the same page:
- If you or your chapter intend on collecting usernames of people who participate in "in person" events (edit-a-thons, workshops, wiki takes your city, etc), then you have to use the opt-in forms. We know that some people might get scared - that is one reason why this opt-in form exists. We have to disclose to them. If they express unease or fear about data collection, we suggest that program implementers (like you, the person who plans and implements the event) explain more to them if needed, and if that works, they can opt in, and if it doesn't, they opt out. Giving a choice at in person events is important.
- Hope that helps just to clarify again. SarahStierch (talk) 19:51, 16 December 2013 (UTC)
- [citation needed] Nemo 07:13, 18 December 2013 (UTC)
- Hi User:CristianCantoro! I do want to clarify, just so we are all on the same page:
- Can you perhaps expand on the point you're implying by adding this citation needed tag, Nemo? It's not very helpful on its own. Jtmorgan (talk) 21:43, 18 December 2013 (UTC)
Wikimania evaluation
[edit]I am curious, how is Program Evaluation planning to instrument Wikimania and evaluate Wikimania this year? Also, will Program Evaluation provide a guide to Wikimania participants about how to maximize the "impact" of their participation? Some informal discussion about how to get value out of the Wikimania hackathon has already happened on the tech mailing list. Thanks, --Pine✉ 19:16, 23 July 2014 (UTC)
- Hello Pine. Somehow missed this question. We have been working with the organizers of this year's Wikimania to develop an exit survey similar to that which was done following the Wikimedia Conference in Berlin (View their very recent reporting and link to survey data here). Similarly, the Wikimania survey targets goals of learning, networking and collaboration and will include measurement of the conference content overall; the organization and logistics of the conference; evaluation of the themes and their related content presented in terms of usefulness; participant networking and collaboration; overall learning outcomes; participant satisfaction and basic background demographics. The items are currently under final review and comment with a number of current, past, and future Wikimania planning stakeholders. This survey will also link to an optional set of items to assess survey-based outcomes for hackathon participants as well. While also still in the planning stage with the planning team, those items will likely include measurement of perceptions about the structure of hackathon days and projects, opportunities for mentoring, guidance, and collaboration and the usefulness and outcomes and/or potential outcomes of those opportunities. In addition to this exit survey, Ed Saperia is arranging for participation counts to be taken at each of the sessions and pushing to have people share lessons learned and hot conversations via twitter to the conference hashtag for potential content analysis. JAnstee (WMF) (talk) 19:24, 28 July 2014 (UTC)
- Thanks User:JAnstee (WMF). This is good to hear. Will you also be evaluating the online impact of Wikimania particularly as it relates to active editor statistics and editor diversity? Also, would you be willing to adapt or create a survey to evaluate the Research Hackathon next week? If you can draft a survey I can review it with the other event organizers. It may be possible for us to simply adapt the dev hackathon survey for the Research Hackathon. Thanks, --Pine✉ 07:32, 31 July 2014 (UTC)
- That is a good follow-up point. As we do not directly evaluate, but work to support self-evaluation of programs, I cannot speak to those goals and related evaluation strategies except for the portion of the conference for which organizers have shared goals for direct online impact, the hackathon. For the hackathon days, organizers plan to track the product of hackathon efforts in terms of the bugs fixed, templates created/added, analyses run, citations added, and the like, however this tracking is the responsibility of the organizers. As for tracking online activity of all Wikimania conference participants, that would be possible with usernames, and Wikimetrics at appropriate follow-up points, however, since the primary goals of Wikimania have not been stated in terms of intermediate behavior outcomes specific to online activity, there is currently no plan for that at this time, only immediate outcomes and short-term learning, networking, and motivation outcomes as targeted by the existing survey draft items. As for that draft, I would be happy to also send your way for input as well (I will share that draft to your inbox). JAnstee (WMF) (talk) 17:34, 31 July 2014 (UTC)
- Thanks User:JAnstee (WMF). This is good to hear. Will you also be evaluating the online impact of Wikimania particularly as it relates to active editor statistics and editor diversity? Also, would you be willing to adapt or create a survey to evaluate the Research Hackathon next week? If you can draft a survey I can review it with the other event organizers. It may be possible for us to simply adapt the dev hackathon survey for the Research Hackathon. Thanks, --Pine✉ 07:32, 31 July 2014 (UTC)