Learning and Evaluation/Archive/Connect/Questions/Archives/2013-10

What are the established Goals and Metrics used to evaluate Hackathons

WM.IL is planning a hackathon in TLV and would like some input from the organisers of previous hackathons. For example:

1. Costs of running these type of event.
   • Venue, Internet and catering costs.
   • How much is typically allocated for participation and for Staff.
2. Key Indicators of success of this type of activity.
   • Metrics of participation ?
   • Long and short term participation change ?
   • Bugs fixed.
   • New account creations.
   • Code/Media created.
   • Sprints on current projects.
   • Press & Social Media

Since several hackathons are budgeted annually by WMF and chapters - where are budgeting of these events are documented?

Thanks OrenBochman (talk) 18:49, 7 October 2013 (UTC)

Hi Oren! I don't have a simple answer for you re: key success metrics. One of our goals in creating this portal is to begin to develop those metrics collaboratively and transparently. I will think on that a bit today, however, and try to provide some suggestions later on. For now, if you're looking for logistical info such as budgeting, the best place to look is in the reports from previous hackathons, most of which are listed in the Grant report index and categorized under Category:WMF grant reports. You can filter these reports for relevance by doing an advanced search for "hackathon" in the Grants namespace[1]. Cheers, Jmorgan (WMF) (talk) 19:04, 7 October 2013 (UTC)

As far as I know we don't have any standard document to help you answering these questions. Most of the experience accumulated after a tech event doesn't reach future organizers and it would be very useful to solve this problem. Then again we do have experienced organizers kin the community. How many participants are you expecting? For an event of about 50 people the mw:Bangalore DevCamp November 2012 is a good reference. For bigger events then you should look at mw:Amsterdam Hackathon 2013. I haven't attended these events though, and I don't know about their budgets. Also, the costs will depend obviously on your local context. Not only because of local prices but also about local expectations: T-shirts expected? Are sandwiches ok for lunch? Do people expect coffe and a wide assortment of cookies and fruit during the breaks? A night event at the roof of a luxury hotel with all drinks covered?  ;) Personally I'm for Spartan tech events, putting the budget in speed WiFi and sponsoring travel for great community participants.--Qgil (talk) 18:11, 8 October 2013 (UTC)

Thanks for tips, Quim. Oren, I've been thinking about this a bit more, and here are a few ideas:

• Accounts created, bugs filed/fixed, documentation written, and code committed at the event are definitely things to track. In general, you probably want to track the kinds of contributions that you feel are most valuable to the project.
• Definitely make sure you document the usernames of participants. Get other contact info if you plan to follow up with people off-wiki (for surveys, invites to mailing lists and further events, etc).
• In terms of tracking long-term impact of the event, it could also be useful to look at subsequent edits for the cohort of people who take part, particularly newly-created accounts. I believe that the Wikimetrics tool is currently set up to provide several different useful measures of participation over time.
• A feedback survey can be useful for figuring out what people liked, what they want to see next time, to find out a little more about who your participants are, etc. There is a small selection of survey-related tips here, and a (way too long) how-to document here.
• don't forget to take pictures :)

In terms of event planning and logistical stuff (which Quim began to sketch out): today I pulled the "Lessons learned" sections out of a bunch of recent grant reports and aggregated them here. Browsing through these could give you some ideas for things to try out, and pitfalls to avoid. Unfortunately, none of these event reports are for DevCamps or Hackathons, but a lot of the considerations are probably generally-applicable to Wikimedia events.

Hope some of this helps at least a bit! Cheers, Jmorgan (WMF) (talk) 21:27, 8 October 2013 (UTC)

The event in planning like the one we reported on at http://blog.wikimedia.org/2013/05/29/first-wikimedia-hackathon-in-tel-aviv-israel/ which may host 30-60 hackers. I'm think that if WMF engineering's has some reports on hackathon, it is time that these are opened up. Particularly since these are financed from the same contributions FDC and PED is oversighting.

Here are a number of issues I am concerned with.

1. unlike editing session hackathons do not require any participation on wiki.
2. Installing and configuring a modern mediawiki version requires:
3. [LW]amp (apache,mysql,php)
   1. a cache
   2. git & gerrit,
   3. mediawiki,
   4. extensions,
   5. parsoid and nodejs
   6. lua
   7. JVM + the search engine
   8. Puppet

The above componnents need to be configured as well and this is incredibly complex and time consuming. WMF recommends two installs - test and development. Finally importing data is very time consuming and complex. I've never been to a hackathon in which half of these have all been demonstrated...

1. there are not stats on how many first time hackers make use of get gerrit, particularly since there is a significant learning curve.
2. we ask as many people as possible to make their get/gerrit userids before the hackathons since setting these up is very time consuming for limited available personnel.
3. Like cost of other WMF sponsored hackathons, these are required for a baseline for reporting.
4. I like to give people t-shirts as most hackers will attend multiple hackathons - and end up promoting our events. However unless there are some suggestions - I don't know how to measure this kind of impact. What is PED position on project components that are harder to track like this one ?

Legal Issues

1. Unless an WMF tool like wiki metrics is developed for tracking cohorts hacker activity we will face legal issues related to for tracking individuals, (privacy and database laws). While a representative of the WMF legal team has expressed a view that local legal issues require attention and compliance per local requirements - we are not getting the required support from WMF legal department.

I am reluctant to involve them and would not expect a timely response (before the hackathon) on how we can comply with Israeli laws in this area as well.

1. Safe space policy - How have breaches to the safe space policy been handled in recent events? I've noticed that these too are are not being reported.

Oren, I don't know whether there are reports for other WMF-sponsored hackathons available. I would expect these to be publicly documented (like all of our reporting) somewhere on MediaWiki. Perhaps Quim can point you to a category or index page for these reports, if such a thing exists?

I don't think you need to demonstrate the long-term impact of t-shirts :) Seems like a pretty tried-and-true incentive/PR strategy (I'm wearing one of my many, many conference SWAG shirts as I write this).

I posted several of your other questions on Wikitech-l yesterday, but have only received one response so far: "Please understand that one of the most important aspects do not get caught in numbers.. Particularly the social aspects are very important. The fact that people meet is often enough to get to new levels of understanding, inspiration. This is the reason why people travel all over the world... to inspire and they do. Thanks, GerardM."

As for potential Israeli legal issues with tracking cohort hacker activity (commits? bug reports?), that's far beyond my expertise. I don't know whether using a WMF-blessed analysis tool would provide additional coverage, our dev logs are already public, aren't they? Forgive my ignorance on the matter.

Again, if anyone reading this has advice or information to share, I encourage them to post it here. Jmorgan (WMF) (talk) 21:21, 23 October 2013 (UTC)

Hello Oren, WMF legal team cannot advise you on the privacy rules for your local event. There may be ways for you to collect information about your program by only collecting anonymized and public data. I would encourage you to be respectful towards participants, and only collect information with an individual's permission and in a manner consistent with Wikimedia values. Ultimately, each event organizer is responsible for ensuring that they comply with local law to the extent that they are required. Thanks, Stephen LaPorte (WMF) (talk) 00:10, 29 October 2013 (UTC)