Grants talk:PEG/WikiSym Initiative/2015 OpenSym Conference

A decision has been made on this grant. Comments are still welcome on this page.

GAC members who support this request

1. --DerekvG (talk) 14:48, 31 January 2015 (UTC)

GAC members who oppose this request

1. I am not really sure about it but it seems that the overall cost is in charge of this grant (except the fee of the attendees) to have a conference about open content and open software where the real impact in Wikimedia projects has not a clear view. In my opinion the best would be to find a co-financing partner and it will help also to communicate better the results of this event in order to have more outreach. Sorry if I am so strong but an recurrent event, which is enable to find external sponsors and is still looking the full financial support within WMF, is going in the wrong way, probably the problem is the bad communication. Regards. --Ilario (talk) 11:08, 9 February 2015 (UTC)

I hope it is appropriate to comment here directly. If not, please move my comment.

Could you please clarify how/why we communicate "badly"? Given our recent change from WikiSym to OpenSym when I look at our traffic and the downloads and the free proceedings at http://opensym.org/archives I'm thinking we are doing reasonably well. Of course we can always improve and I'd like to learn how. Indeed, we want to grow. Dirk Riehle (talk) 20:51, 9 February 2015 (UTC)

We are not asking fur full support; we are asking for partial support in the same amount like last year. Dirk Riehle (talk) 20:51, 9 February 2015 (UTC)

GAC comments

Evidence

Considering that this OpenSym Conference is not the first edition, I would clearly know the impact received by the Wikimedia movement in the past years. regards. --Ilario (talk) 08:54, 27 January 2015 (UTC)

You could probably read it here: Grants:PEG/WikiSym Initiative/2014 OpenSym Conference/Report, however it is so vague in answering question about real impact that it is hard to evaluate the real effect of the conference on Wikimedia projects. Therefore some questions:

1. You claim "Option B: How did you improve quality on one or more Wikimedia projects? We are one step removed from immediately improving a Wikimedia project. The research presented at OpenSym helps improve projects, but does not do it itself. Also, simply by having an outlet like OpenSym, researchers can get published and are motivated to perform this research; without OpenSym, there would be less researchers performing research into Wikimedia projects." - can you show at least several good examples? I mean - show briefly how the research presented in 2014 at least could help to improve quality of Wikimedia projects?

Collecting this evidence, which we have done partially in the past, is a lot of work. Let me try to do it the following way: I'll send out email to past participants and ask them to provide their input on whether they research work did any good? Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

1. Assuming that there are good examples how the research could help to improve quality of Wikimedia projects - have anyone checked if the quality was indeed improved? How and in which areas? (Both questions can apply not only to the research presented in 2014 but also in previous years.) ?
2. How many papers were released under free licenses in 2014? You claim 64 papers published - 20 about Wikipedia - but do not count how many under free licenses?

Please see the discussion on Open Access Publishing on the grant proposal page. About 5% of the papers in the last year were published under a free license, because those authors could afford to pay the fees. I communicated with our publisher (the ACM) for a package deal which would allow to have developing-countries authors to participate in open access, but they won't do it (yet). We would have to pay for every paper indvidually (US$ per paper, among the cheapest in the industry) but still prohibitively expense. Please note that I care deeply about open access publishing but have to walk a tightrope of researchers wanting the ACM digital library, not being able to afford the OA fees, and pushing the envelope. I'm currently working with PeerJ for a much cheaper deal but might still get rejected by authors. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

So, how many of those 5% were devoted to the Wikimedia projects? Polimerek (talk) 11:18, 7 March 2015 (UTC)

1. Regarding budget: last year you have asked for 15 000 USD - and the rest (out of 40 000) were claimed to be donated from other sources. So it was 37.5% of the budget. This year you ask for 20 000 USD out of 30 000. Why so much?

We asked and received EUR 15000 last year, which is the same we are requesting this year USD 20000. We were asked to submit soon so the budget isn't fully stable yet. Please see my overall comments below on the budget. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

Polimerek (talk) 12:21, 29 January 2015 (UTC)

This is a long standing event, and I don't want to question relevance of basic research, however my I suggest that you also do a "round up"-project that targets all the past participants, their initial proposal and contributions, and the ultimate result (impact) . for me the ultimate result or impact of a project is not that a proposed project got implemented (perhaps with a few changes) but what did reach the end user and got implemented a a result of the initial proposal even if it wast the complete opposite. An apple could have been initially proposed and we ended up with a bowl of fruit punch or the proposal was a new electric car and we end up in reality with a bathtub on wheels ... it might be interesting to collect and present that information however I don't think its necessary to collate that information as a motivation for this project --DerekvG (talk) 14:59, 31 January 2015 (UTC)

This is a great idea. We will think about how to do it. Right now we are too busy rebuilding/growing after the shift from WikiSym to OpenSym but we should do it exactly for the purpose you mention: To demonstrate, by evidence, the good we did. Not sure how quickly we'll get this implemented, but we'll try. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

We collected "evidence" by asking past participants through a mailing list whether they thought WikiSym/OpenSym had been useful and how. There answers are here: Grants:PEG/WikiSym_Initiative/2015_OpenSym_Conference/Evidence. This also lead to a couple of more endorsements on the main page of this grant proposal. Dirk Riehle (talk) 16:48, 10 March 2015 (UTC)

Budgeting

• There was one question about budget above, but I would also add some more comments: you want participant to pay entrance fees - how much money are you expect to collect? Why those amounts weren't included into the budget? Are you really sure that anyone will pay for the participation? rubin16 (talk) 16:00, 31 January 2015 (UTC)

I don't fully understand the grant proposal form but it really only asked about expenses, not revenues. I'm not sure why, I would indeed have asked for revenues as well. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

• could you please provide more detailed budget breakdown? What equipment do you need? How catering cost was calculated? you have food in "event location and transportation" and "catering" below. Facilitator for 1 ths USD? Really? How many hours do you expect him to donate to the project? I wouldn't agree with such costs without detailed explanation per people, items to be bought and the cost of each item rubin16 (talk) 16:00, 31 January 2015 (UTC)

I will replace this line with more information in a few days after returning from a trip to the event venue. Please stay tuned. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

Questions from Rebecca

Hi Dirk! Thank you for the proposal! I've been following the Opensym privately for some years (though I never attended) and really do like it! I do have a couple of questions regarding the success measurements for the conference tough.

1. Like DerekvG I would like to know more about the impact this conference has inside and outside the Wikimedia movement. Especially since it's been running 11 years now and with open access, crowdsourcing etc on the rise I would expect there to be awareness of the conference outside of those attending.

This and the next bullet item are great questions; I think I will have to crowdsource the answer by going back to participants. Please give me some time to do that.Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

Sure! This is after all a volunteer project, so take the time you need. :-) --Rebecca Cotton (talk) 21:24, 18 February 2015 (UTC)

We now got some answers, pointed to above under Evidence already. We collected such here: Grants:PEG/WikiSym_Initiative/2015_OpenSym_Conference/Evidence Dirk Riehle (talk) 16:49, 10 March 2015 (UTC)

1. Apart from meeting each other and networking what impact does the conference have on the work of the attendees? Do you know of any collaborations the conference helped to foster? Since you make the accepted papers openly available, did you see this and the attention from the conference having any effect for the reach of the papers/research in terms of downloads/reads, mentions and citations?

Influence is difficult to measure and sometimes very subtle, but in this case we do have a claim to fame. We defined the notion of open collaboration, decided to rename the conference accordinglly. This reframing and focussing seems to have stuck, as we see more researchers pick up the term. An example is a broad US/European collaboration effort, the open collaborative data factories initiative.

1. Do you have numbers on diversity of the attendees of the conferences? Male-Female-ratio and what regions they came from? Do these criteria play any role in accepting talks?

   We don't have/don't collect such data. There is no affirmative action in science, so diversity criteria should not have an impact on whether talks get accepted. However, the industry and practitioner track is fairly easy to get into (if you don't mind having your work not "branded" as peer-reviewed academic work).

   Well we know it 'should' not make a difference, nonetheless it would be interesting to see if it does. But if is does make a difference there are fairly easy ways to work around such discrimination without taking affirmative action, e.g. anonymizing the submissions before they are reviewed. Since a few publishers and conferences have found a discriminating bias when selecting submissions this might be interesting for you look into. :-) --Rebecca Cotton (talk) 21:24, 18 February 2015 (UTC)

2. Since from what I see happening at other conferences, looking at the topics that are on the rise, etc. I would hesitate to call a stagnating number of attendees and paper submissions a success for this conference. With the conference being held in San Francisco do you have any reason to believe you will not be able to get 150 participants? Is there some criterion that thins out the research crowd so drastically that this growth isn't possible?

   This is a hard question. There is simply a glut of conferences and journals, so researchers have a broad array of publication outlets to choose from. We hope that SF will be an attractive place to travel to (in addition to wanting to be at the conference, but we honestly don't know yet how it will play out.) But I'm sure the growth is possible! Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

3. The same goes for the number of papers submitted. Excluding the first two conferences you have on average 47 papers submitted with 64 at last years conference at 94 attendees. Assuming a goal of 150 attendees with an attendee/submission ratio somewhere between 30%-65% with 66% for the last conference would you assume 100 papers an impossible goal?

   This is an on-going debate as to how selective "high quality" a conference should be. I prefer a community conference over a super-selective conference. Ultimately, we want people to come for the community, exchange, collaboration, inspiration. Ideally, we have a large number of people who attend without having a research paper published. We are not quite there yet. Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

   Selective in regard to their submissions or their attendees? If you say your goal is to have more people attending without published papers, but have not quite reached that goal jet, have you in anyway changed your strategy to get more attendees, even if they have not submitted their own papers? Did you experiment with your communication strategy, and do you maybe have a lesson learnt of two for us what has and has not worked in the past years? --Rebecca Cotton (talk) 21:24, 18 February 2015 (UTC)

Realistically, not yet. Most conferences never get to do this, because they are "thrown over the wall" from one year to the next (different organizers) with very little learning in between. We are trying to change that by making my university and our strong admin support be the backbone so that in future years a more strategic conference chair can experiment with strategies while my group holds down the fort. This year I had to step up as the main organizer, keeping me busy on both fronts, general outreach and admin matters. I'm hoping for next year! Dirk Riehle (talk) 16:48, 10 March 2015 (UTC)

Please do not take these questions as criticism of the amazing and continuing work you are doing! I am just trying to get a better understanding of what kind of space the conference is operating in, what kind of roadblocks you may have, etc. All the best from Berlin! --Rebecca Cotton (talk) 19:53, 8 February 2015 (UTC)

Thank you for the kind words! We are working hard and things are rarely perfect. Your encouragement and critical questions were very welcome! Dirk Riehle (talk) 20:53, 9 February 2015 (UTC)

Lookign forward to hearing from you again, and what you found out! Best! --Rebecca Cotton (talk) 21:24, 18 February 2015 (UTC)

General clarifications on budget

I apologize for the delay; I was scouting social event locations for OpenSym in San Francisco last week, now traveling back to Germany.

Some general comments first, which broadly answer to several questions:

1. This is a mostly a research conference; we are working hard to attract industry / practitioners, but ultimately, we remain researcher dominated. In terms of participation fees, research conferences fall somewhere in the middle. They are not commercial (fee > $1000) and they are not fully subsidized practitioner conferences (fee < $100). We are a non-profit and budget to break even with the goal of being affordable to most. Expected fees are $200-$250 for PhD students and self-paying participants and $400-$500 for researchers and professors.
2. The budget calculation in a way is much simpler than the requested breakdown suggests: We have a total cost X; we reduce it by the sponsorship we get (Google and Wikimedia in the most recent past), and adjust the participation fee based on the remaining uncovered costs. Thus, depending on how much funds we can raise, participation fees go up or down. Because participants are on (some) budget, it is possible to charge the fees given above.
3. The major unknown in the budget, hopefully settled soon, is the 2nd-day social-evening event. San Francisco is really expensive! It seems hard to stay below US$ 100 per person (people usually laugh now when I say this). Please see the hard work we put into finding affordable venues. Recommendations welcome! Some conferences exclude cheap-fees-PhD students from the social event and we really don't want that. The Wikimedia contribution helps ensure that PhD students remain an integral part of the OpenSym community.

Dirk Riehle (talk) 20:56, 9 February 2015 (UTC)

On #3, check your email for some venue ideas. -Pete F (talk) 04:22, 19 February 2015 (UTC)

Thanks, Pete!! Dirk Riehle (talk) 16:50, 10 March 2015 (UTC)

Past funding

Hello Dirk,

I am interested in the list of past funding of this event. Was WMF sponsoring it too prior 2014? Who were other sponsors? What was the ratio of individual funding parts on the total funding? Some table like Year | Source(*) | Amount | Percentage together with amount spent for each year would be nice. (*)=Sponsors, entrance fees, in-kind donations... I was trying to find it on your site, but no luck. Could you please provide it either there or here? Thank you.

— Danny B. 23:52, 4 April 2015 (UTC)

Past composition of attendees

Hello Dirk,

I am interested in the list of composition of attendees from past events. Basically their affilitaion group (aka no need for particular company or organization, rather things like "university professor", "student of economy", "IT sector company employee", "NGO volunteer" etc.). How many Wikimedians were involved? Thank you.

— Danny B. 23:52, 4 April 2015 (UTC)

Community comments

Remarks from Pete Forsyth

I have presented at two WikiSym conferences (see my 3 entries on the "evidence" subpage), and I believe it is an important event. However, I think there are a couple of critical (but correctable) problems with this grant proposal:

• The #Fit with strategy section is too thin, and it does not connect with the #Measures of success section. "Improving quality" is about improving the quality of Wikimedia's content offerings; but it's hard for me to see how an academic conference could have much direct, or clearly attributable, impact on that. I think reach or participation might be better fits, and it might be easier to make a case. "Measures of success" is, I think, the place to make that case; but the things measured here have nothing to do with the "quality" goal stated.
• Dirk states above: "We would have to pay the USD 700 for each paper that the author also has to pay. This would completely break the conference finances, so we decided not to do this." But this proposal is, in part, about determining the conference's finances, is it not? So this judgment seems premature. The paywall of ACM is, IMO, a major problem both on philosophical level, and in terms of maximizing the practical impact of any money invested. A little math to illustrate my perspective:
  • Goal of 20 submissions related to Wikipedia; let's (arbitrarily) assume half of those are accepted, so:
  • 10 Wikipedia-related papers published by ACM
  • Multiply by $700 per paper for free licensing, so:
  • For an additional $7,000 (increase of 35% over the requested $20k), we could have an outcome of 10 freely licensed and peer reviewed research papers about Wikipedia.

As I see it, that $7,000 investment -- which could support intellectual and social connections worldwide among those committed to (or at least interested in) Wikimedia's goals -- would yield far greater results (at least, on the "participation" goal) than the $20,000 that merely brings a 100 people located near San Francisco, and capable of paying $200 to $400, together to forge stronger connections.

So overall, my recommendation would be to explore the strategic goals/fit more carefully and explicitly, and to increase the ask by whatever amount would permit publishing all Wikimedia-related publications produced under a free license. -Pete F (talk) 00:14, 18 February 2015 (UTC)

Tony1

Thanks for this application, Dirk. A few comments:

1. I agree with other comments about the high cost of catering (and my view is that people should pay for their own alcohol). $80 per person for a social event is rather steep, and might carry a co-contribution from participants.
2. Please provide more details in the budget. Number of nights and number of people at ?three-star hotels, is it? Flights from where?
3. Rounded figures suggest that you'll underspend and return some of the funds to the WMF—is that a reasonable assumption?
4. SF is an ideal location.
5. I think we deserve to know who will be paying the other $10,000, and in the budget how much the WMF will be paying for each item.
6. Measures of success, and risks: I appreciate the attempt at humour through exaggeration; but one of us did die at the Berlin conference a couple of years ago, so perhaps this could be excluded from any future application.
7. Will all outputs be published with open-source licences?
8. Tenuous link with the WMF's priorities and impact on its sites. For this reason alone, the ask should be more slender—like about half or less, not two-thirds, of your total. Tony (talk) 03:36, 5 April 2015 (UTC)

WMF comments

Hi Dirk. Thank you for this request and engagement in the discussion. We really appreciate your efforts to source feedback on the impact of past events. We have a number of remaining questions on the request:

1. Measures of success: A post-conference survey would help us better understand the immediate impact of the conference -- participant satisfaction with the program, presentations, networking opportunities, etc. Another survey 6 months post-conference would also give us an understanding of what collaborations have been initiated, and research developed or published as a result of the conference. Continuing to solicit feedback in the manner that you have been doing on the Evidence page will help build a stronger case for the impact of the conference year-on-year. Our Learning & Evaluation team has number of survey resources and can also discuss other ways to measure impact for this type of event.

I agree on the importance of process learning and follow-up; it is only hard to do once the event is over. Are you offering to help? This would be so much appreciated! I'd have no hesitation to let you get to such data (as long as it is clear to survey participants) and would be happy to work with you after the event to collect that data. Dirk Riehle (talk) 21:16, 11 March 2015 (UTC)

1. Social event: The $7,000 price tag for the social event is very high -- 35% of the entire budget. Have you been able to secure a more reasonable venue?

No, and I have little hope by now. The venue is the smaller part; the biggest part is the expected food. The best offer right now is a $60/person buffet food. That's San Francisco for you :-( I'm sure this will show up under room for improvement on a post-event survey... Dirk Riehle (talk) 21:16, 11 March 2015 (UTC)

1. Please provide more details on how the specific budget items were calculated. For example, what is the breakdown for the catering? You can add these details in the notes section.

For the cheapest option, I'm still waiting for beverage numbers in addition to the food number given above. Will hopefully be back soon. Dirk Riehle (talk) 21:16, 11 March 2015 (UTC)

1. We appreciate the continued discussion around licensing and the issues raised by Pete above. However, as discussed on last year's grant, we believe that having the materials freely (gratis) available on opensym.org and supporting this community of researchers is important enough to justify the investment. We do support making more wiki-research open access, but it is not practical, sustainable, or scalable for us to pay the additional fees to freely license individual papers.

My guts feeling is that we are close to the breaking point and the ACM will give in. As a non-profit (the ACM) I expect a retroactive application of an open license. We should also use a post-event survey to inquire about our community's willingness to let go of the ACM digital library now given an increasing number of reasonable alternatives. Dirk Riehle (talk) 21:16, 11 March 2015 (UTC)

Looking forward to your responses. Alex Wang (WMF) (talk) 23:59, 9 March 2015 (UTC)

Will be back, please give me a few more days. Dirk Riehle (talk) 21:16, 11 March 2015 (UTC)

Here is a detailed expenses listing. I still don't know about beverages for the social event evening but I guess a bit more or a bit less is not the issue here. This spreadsheet should give you a more detailed overview than the Wikitext table. Dirk Riehle (talk) 15:28, 14 March 2015 (UTC)

Hi Dirk Riehle. Thank you for sending the expense details. We are still concerned that the social event is over half of the budget. I have asked the Wikimedia Foundation staff who are experienced in event planning if they have any other recommendations. Will get back to you soon. Alex Wang (WMF) (talk) 22:31, 16 March 2015 (UTC)

Hi Dirk Riehle. I checked with the event organizers at WMF and they said that the venue is quite reasonable (and a great space!). However, the food costs seem quite high for the social event. We typically pay $25-30/person for food for dinner. Have you checked out all the recommended caterers on their site? It looks like they don't require you to use their preferred caterers, but it is recommended. If none of those listed fall in the $25-$30 range, we can provide some recommendations. $20/person for drinks is reasonable, but I noticed that this is still to be confirmed on the spreadsheet. Have you been able to confirm this yet? Sometimes, drink providers will increase the costs dramatically. If that's the case, I would recommend setting a 2-drink limit at a certain price (not over $20/person). Alex Wang (WMF) (talk) 23:28, 18 March 2015 (UTC)

Hi Alex! The venue is great but comes with a required caterer for weekday events. Sadly we can't get a lower price point for dinner. We confirmed the drinks at $20/person (about 1/4 of a wine bottle or beer and two non-alcoholic beverages). We are looking at staying at the main venue for the social event too now which would lower cost a little bit, but not much. (Not yet confirmed that we can extend far into the evening.) Dirk Riehle (talk) 14:30, 23 March 2015 (UTC)

Hi Dirk Riehle. Thanks for the update. Let me know when you confirm if you can stay at the conference venue or not. Alex Wang (WMF) (talk) 17:06, 23 March 2015 (UTC)

Sorry for the delay! After a fair bit of negotation, we are now staying at the main venue for the main conference dinner, which is the cheapest option available. I updated the costs spreadsheet by creating a new tab with the revised costs. Thanks for your patience! Dirk Riehle (talk) 16:52, 25 April 2015 (UTC)

• It says "A decision has been made" at the top. Where is this decision announced? My questions and those of Pete Forsyth have been ignored. Tony (talk) 16:19, 29 April 2015 (UTC)
  • While I'm surprised that the grant wasn't adjusted to make a better connection to the strategic goals (which wouldn't have required any changes to the substance of the grant), I'm satisfied that my thoughts were heard, and respect the decision. (Tony1 the grant was funded.) -Pete F (talk) 18:02, 29 April 2015 (UTC)
    • Then could the invitation to make comments, at the top of the page, be removed or clarified, so that reviewers don't waste their time? Tony (talk) 03:53, 30 April 2015 (UTC)