Grants talk:IEG/Open Access Reader

For comments about the project itself, please comment on the Open Access Reader Project Talk Page. This page is only for discussions about the IEG grant application, and not the project itself.

Some of this was moved from Grants_talk:IdeaLab/Open_Access_Reader

IEG proposals - March 31

Hi, Ed. Glad we got to discuss this idea a bit at the IdeaLab Hangout last week! Just a reminder that if you're still interested in having this considered for an IEG in this round, you'll want to click that "expand" link in the purple bar at the bottom of your page, and follow the steps to create your full proposal by March 31. If you run into any snags with the templates, feel free to ping for help. Cheers, Siko (WMF) (talk) 22:50, 19 March 2014 (UTC)

First Draft of Project Plan

Just jotting down bulletpoints

Scope

In 2013 over 500,000 research papers were released open access in the UK, more than ever before. Following further legislation both in the UK and the US, and continued success of the Open Access movement within the scholarly community, this number will only continue to rise.

Academic research is generally considered the best content humanity produces, literally the cutting edge of human thought. While any given paper may be trivial, if we find a reliable open metric of significance such as academic citation and rank every paper across all disciplines, the top 1% of cited papers are likely to be very interesting indeed.

This project proposes to create an automated process to find the most significant open access research and deliver it to community members in a convenient format to integrate into Wikipedia articles, with the goal of significantly improving coverage across a wide range of high impact topics on an ongoing basis.

We also believe this may be a significant tool for outreach; it gives new editors a never-ending task list of small but interesting tasks, and can be filtered by subject area, so e.g. may be interesting to universities to help encourage undergraduates to do scholarly research.

Activites

• Socialise concept amongst community members and OA experts.
• Develop and iterate specification.
• Identify & recruit engineering talent.
• Build prototype in 8 sprints.
• Promote and support project output.
• Record subsequent activity.

Budget

• Contractors for 8 development sprints at $1500 each
• 6 months product management, HR, community management, customer service, research & development $13000

Potential communities

Depends on input of OA papers, so at first will probably be biased towards scientific topics, but as the OA movement crosses into arts and humanities, so will OAR, hopefully without any additional development work. Example groups:

• Wikiproject Medicine

Fit with Strategy

• Improve Quality
• Improve Participation

Sustainability

• Stability of aggregator API
• Promotion
• Support

Measures of Success

I propose to do this via milestones to begin with, until a functional MVP exists:

1. Identify best Open Access Aggregator
2. Identify and assess available open metrics to create specification for significance filter.
3. Produce a system that generates an up-to-date list of most significant papers.
4. Assess quality of topic metadata to create specification for paper-to-keyword filter.
5. Build and demonstrate paper-to-keyword filter.
6. Configure automated report of most significant papers for a particular keyword
7. Create feedback process
8. Introduce the report to sample communities via Wikiprojects, lists.
9. Report on possible areas of further development discovered during the process.

Then we can track citations added as a results of this initiative, which is a concrete numerical metric.

First Monday article

What's the title of the First Monday article you're citing? Unfortunately (crappy website!) that now just redirects to their front page... Jodi.a.schneider (talk) 08:40, 1 April 2014 (UTC)

The article is http://firstmonday.org/ojs/index.php/fm/article/view/1624/1539%7C, I've amended it in the text. EdSaperia (talk) 16:12, 1 April 2014 (UTC)

"demonstrate deliverability"

I'm not really sure what "demonstrate deliverability" means -- is there another phrase that would fit there? Jodi.a.schneider (talk) 08:42, 1 April 2014 (UTC)

Operationalize the target audience?

The target audience seems spot-on (English Wikipedia contributors in WikiProjects where there is a pressing need to develop a density of citations in order to improve article quality. & English Wikipedia contributors in topics where there has been a persistent lack of success increasing the number vital articles as opposed to featured or good.). Have you come up with a way to operationalize this? Which will be your first targets? Jodi.a.schneider (talk) 08:47, 1 April 2014 (UTC)

Hi Jodi! Part of the task will be doing an overview of which wikiprojects are most active and receptive. Then we'll see what kind of results the OA process throws up, and match them to wikiprojects that would find them relevant. There may also be other ways to find editors by interest besides just Wikiprojects. Exploring this is in the scope of the project EdSaperia (talk) 21:29, 10 April 2014 (UTC)

Developer sprints

How will you identify potential developers? Do you have any potential developers already in mind? Finding appropriate people in the right timeframe could be time-consuming so there is some risk here.Jodi.a.schneider (talk) 08:49, 1 April 2014 (UTC)

Measures of success

I like your measures of success, but I think it would be helpful to say what constitutes "engagement". Also, for the article criterion (At least 50 articles show improvement with relevant content supported by the new citations), what is the timeframe? Will this be possible *during* the grant period, or should it be measured at some interval *after* the grant period? Jodi.a.schneider (talk) 08:51, 1 April 2014 (UTC)

Hosting plan?

Hi Ed and team,

I see you've got a hosting dependency for this project. Please have a look again at the eligibility requirements for these grants. I understand that you're planning to build tools that make API calls rather than building extensions or anything else that requires integration with Mediawiki, and that sounds like a good thing! But we'll also need to see that you've got a hosting plan for your software, and it can't be WMF. What happens after the 6-month IEG ends, and we're not longer providing funding for your hosting solution? I'd like to be able to mark your proposal eligible for review in this round, so hoping to hear from you as soon as possible this week.

Best wishes, Siko (WMF) (talk) 00:43, 8 April 2014 (UTC)

Hi Siko, WMUK have informally pledged to host this project. I will try and get an official pledge. EdSaperia (talk) 21:29, 10 April 2014 (UTC)

I have to say though, I kind of assumed that I'd be able to host this on Labs, and I'm not sure why that's not okay? EdSaperia (talk) 14:21, 11 April 2014 (UTC)

"Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers. Tool Labs is a part of the Labs project, which is designed to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure, including MediaWiki, and to do analytics and bot work. Tool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets ‘orphaned’ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding." https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help - As I understood it, Tool Labs was created exactly to support projects like this. EdSaperia (talk) 11:10, 12 April 2014 (UTC)

I took the line out of the budget. EdSaperia (talk) 11:15, 12 April 2014 (UTC)

Thanks for the update. Tool Labs can be used in some cases - if that's your plan then you'd probably want to investigate a bit further to make sure it meets your needs just as it does for other volunteers :) Since your first ask included budget for hosting, and then there was mention of the UK hosting, I'm not sure anyone knew that Tool Labs was part of your assumed plan. Siko (WMF) (talk) 20:37, 16 April 2014 (UTC)

As far as I can tell, the environment is more than ample, and the project fits the criteria for being hosting there. I got confused earlier because you said "But we'll also need to see that you've got a hosting plan for your software, and it can't be WMF.", so I thought that IEG projects couldn't be hosted on Tool Labs for some reason. EdSaperia (talk) 10:36, 19 April 2014 (UTC)

Eligibility confirmed, round 1 2014

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for round 1 2014 review. Please feel free to ask questions here on the talk page and make changes to your proposal as discussions continue during this community comments period.

The committee's formal review for round 1 2014 begins on 21 April 2014, and grants will be announced in May. See the schedule for more details.

Questions? Contact us.

Please note that eligibility is dependent on the project having a hosting plan that does not rely on WMF, and that no other resources from WMF engineering are needed. If your project is recommended for funding, we will work with you to gather more information about your plan before an IEG can be approved. Siko (WMF) (talk) 00:14, 9 April 2014 (UTC)

Other Wikis?

Hi, thanks for sharing your idea. Before all: I have read only this page, not the project page (I'm so sorry). I like the idea to improve the citation sources and content in Wikipedia, but is this possible to do in other Wikipedias?. As I see, the citation sources would work in English Wikipedia because the analysis is written in English, and the semantic will work in English, but I'm a non-english speaker. Just a question "for the future" ;). Kind regards Superzerocool (talk) 01:53, 9 April 2014 (UTC)

Hiya! The main restriction will be the sources of open access papers. If we have papers in other languages, it should be easy to offer the rest of the process in other languages too. For this first application, only English is in scope, but localisation is certainly something we can look at in future rounds. EdSaperia (talk) 21:34, 10 April 2014 (UTC)

Frankly I don't manage to understand what you're actually and concretely going to produce ("deliver a complete workflow"?? is that a thing?), but whatever it is, if it's not generic enough to be reused outside the English Wikipedia it's worthless. --Nemo 07:35, 15 April 2014 (UTC)

Designing and supporting workflows is definitely a thing: https://en.wikipedia.org/wiki/Workflow_management_system. It is generic enough to be used on any Wikipedia, but localisation is out of scope for this phase of the project. EdSaperia (talk) 10:26, 15 April 2014 (UTC)

I know that but it still doesn't explain what it is in this context. mw:i18n is always too late if it's not there since the beginning. --Nemo 10:43, 15 April 2014 (UTC)

feedback for proposal

Hello! I have some questions regarding this proposal, would you please help me to improve my understanding? rubin16 (talk) 06:38, 10 April 2014 (UTC)

• is there any timeline for the project? when are you planning to find developers, when will the database be designed, etc.? rubin16 (talk) 06:38, 10 April 2014 (UTC)

I'll put together a draft timeline on the project page in the next week. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

• can it be somehow expandable to other languages? Or will you framework be targeted only on English-speaking projects? rubin16 (talk) 06:38, 10 April 2014 (UTC)

I think expanding to other languages should be fairly easy, but is out of scope for this first round. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

• I know about SuggestBot on Wikipedia that analyses past contributions of the user to learn his interests - the project itself is available on Labs: have you contacted/discussed possibility of coordination with the bot owner? It seems that bot's abilities are in your scope: "proactive notification or delivery of content to editors beyond availability on project pages or portals - perhaps by developing bots to cross reference user box interest" rubin16 (talk) 06:38, 10 April 2014 (UTC)

Ah yes, I've heard of this - sounds like a great idea, I'll made a note on the project page to follow up! EdSaperia (talk) 22:07, 10 April 2014 (UTC)

I also have a number of questions for budgeting, please, don't take it as I am against the proposal or completely disagree with it, treat is as my professional scepticism obtained in real-life job :) rubin16 (talk) 06:38, 10 April 2014 (UTC)

• developer hours: are you going to hire 1 developer for 160 hours or a number for total of 160 hours? Jodi.a.schneider already asked on this page about already performed search for candidates and I would like to join to her question; rubin16 (talk) 06:38, 10 April 2014 (UTC)

It is eight sprints, each consisting of 20 hours, probably spread over a few days. It'll probably be a several different developers, depending on what the sprint focuses on. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

• have you already performed technical description of the task for the project? Is it possible to see it or just its overview? As I see from Meta page of the project you haven't still selected even an aggregator to be used rubin16 (talk) 06:38, 10 April 2014 (UTC)

Evaluation of different aggregators and developing a technical specification are both within the scope of the project. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

• how have you come to 75$/hour rate for developers? I've performed a research on Odesk.com for developers in sections "scripts/utilities", "software project management", selected those that are from Western Europe, have top scores and already billed more than 1,000 hours from service and received the following structure: 29% ask for lower than $25 for hour, 50% - for 25-50$ and 21% - for higher than $50. Why do you want to hire a UK programmer, have you evaluated possibility of attracting of outsource developer? Even UK programmers seem to be cheaper than your rate per hour... rubin16 (talk) 06:38, 10 April 2014 (UTC)

In the past I've had more success working with local programmers rather than remote, and $75/hr is from experience an indicative rate for a good developer in London. I'll take a look at Odesk.com. Is there an accepted process for hiring contractors in IEGs? EdSaperia (talk) 22:07, 10 April 2014 (UTC)

I am requesting @Sbouterse (WMF): for this question :) rubin16 (talk) 04:40, 11 April 2014 (UTC)

I am in a presentation at WMCON2014 right now, where the Wikidata team is saying that using a local (as opposed to remote) team was a key part of the success of their project. EdSaperia (talk) 14:18, 11 April 2014 (UTC)

I understand this point but I am definitely sure that 75$/hour is a too high rate, so I just proposed a solution. I think that we need a deeper look for this expense: what qualification do you need? is it possible to divide work into some sections where experienced (and expensive) developer will review or consolidate work of juniors who would prepare most part of the work? I can prepare some research here about average rates in Western Europe and the UK but I think you know your project better and it would be easier for you than for me to justify this expense rubin16 (talk) 17:49, 11 April 2014 (UTC)

We don't have set guidelines for rates/hiring process for IEG contractors, but we do use benchmarks from WMF contractors too, when that's useful, to help assess if costs seem reasonable. I agree with rubin16 that a rate of $75 an hour seems high at first glance. When benchmarking, it might be helpful to explore the difference between market rates and non-profit rates, because people often adjust their rates as an act of service when being involved in a project that does social good. We've hired contract developers in San Francisco (not quite as expensive as London, but still!) for less than this estimate, and of course developers in some other countries can be found for much much less, but if you know in advance a local person is needed to accomplish your project, that's an important consideration. If this project is recommended for funding by the committee, with some lingering concerns about the labor costs, we can also discuss this particular line item further in the due diligence phase before finalizing the budget. Cheers! Siko (WMF) (talk) 22:32, 11 April 2014 (UTC)

Ah, interesting, non-profit rates may be different. I'm 100% willing to accept a benchmark SF rate if you have one - are these published anywhere? EdSaperia (talk) 23:24, 12 April 2014 (UTC)

Just to remove any doubt, if you propose a reasonable non-profit benchmark developer rate for SF (or London for that matter) I will update the budget to match. --EdSaperia (talk) 10:02, 25 April 2014 (UTC)

• what will project management particularly included? How much hours would these tasks require? I wouldn't compare proposed rates with average salary as project management doesn't mean full-time working (and average salary means). rubin16 (talk) 06:38, 10 April 2014 (UTC)

I think this is quite comprehensive in terms of tasks: https://meta.wikimedia.org/wiki/Grants:IEG/Open_Access_Reader#Product_manager. I budgeted 380 hours product management = 14 hours a week over six months for equivalent of $36.50/hr, which is comparable to a £40,000 before tax salary in the UK. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

• how will you support hosting after the end of the grant? rubin16 (talk) 06:38, 10 April 2014 (UTC)

I am expecting WMUK to host this project - I am finding out if they will formally pledge to do so. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

Thanks for your answers rubin16 (talk) 06:38, 10 April 2014 (UTC)

No problem :) if you want more detail, please let me know. EdSaperia (talk) 22:07, 10 April 2014 (UTC)

Questions about chapter and community engagement

Hi there,

A few more questions:

1. I'm curious to know why WMUK isn't funding this project directly - it seems like you've got plans to coordinate with them on hosting, etc, and the project is clearly focused on English Wikipedia. Do they prefer that WMF fund this rather than WMUK, and if so why? (feel free to send a chapter person this way to address this instead, if you like...we like to be sure that we're helping and not competing with chapters!).

   WMUK are not currently funding technology projects of this size. EdSaperia (talk) 13:54, 17 April 2014 (UTC)

2. As the main organizer of Wikimania, do you see any potential for conflicts of interest in using that conference as a venue for promoting one particular project which you'd be paid to work on? Curious to hear how you might handle any concerns that could arise around this.

   I am organising Wikimania as a volunteer. I haven't submitted a session about OAR to Wikimania because I suspect I'll be too busy running the event to give it properly. Besides, the programming process is delegated to a programme committee, which has a COI policy. OAR is a non-profit project designed to improve the content of the Wikimedia projects and further the mission; I will certainly be trying to promote its use within the community whether or not I am awarded a grant, so I really hope I can socialise it at Wikimania, but I won't be doing so in any capacity given to me by my organising role. EdSaperia (talk) 13:54, 17 April 2014 (UTC)

3. I expect this project's impact on Wikipedia will rely on having Wikipedia editors from various Wikiprojects actively participating in implementation/use of whatever templates etc you develop. Wikimania and Chapters/Thorgs are good ways to engage the broader movement, but these aren't necessarily the same people who can help you make the idea actionable on Wikipedia (many of those people are quietly sitting behind a screen somewhere, editing Wikipedia!). I wonder if you've already got any Wikiproject editors already waiting in the wings to make use of your tools, or if someone on the project team will have the experience/responsibility to help build these connections for the project?

   I've had contact from Blue Rasberry who suggests Wikiproject Medicine is interested to use this, and from informal discussions with the other OA advisors a number of wikiprojects have previously expressed interest in a tool along similar lines. Community socialisation is something I expect will take up a proportion of my time during the development of the project, and through organising Wikimania I've made contact with many projects and community members. I've also had interest from the Wiki Education Foundation in using this tool in their outreach work. EdSaperia (talk) 13:54, 17 April 2014 (UTC)

Thanks in advance for thinking through what may feel like some complex questions without any "right" answers :) Siko (WMF) (talk) 21:06, 16 April 2014 (UTC)

Some thoughts

This project certainly seems like it has a lot of potential, and would be a great complement to The Wikipedia Library (in fact part of me feels it might make sense to integrate it within that "brand"). Its fit with strategy is strong, and the technical aspects appear ambitious but achievable (though I'll admit to being no expert on these).

My biggest concern is the timing. I'm a great fan of Ed, he has a rare combination of technical knowledge, project management skill, and boundless enthusiasm. However it seems like this project would clash with the running of Wikimania, which is a huge undertaking for Ed (and WMUK). With the size of the Open Access Reader grant requested, and the amount of management that would be required, I'm not sure that it can get the attention it deserves during this time. My recommendation would be to revisit this project in a future IEG round. the wub "?!" 21:59, 19 April 2014 (UTC)

I'll be busy with Wikimania until the middle of August, but while doing it I'll be meeting with a ton of relevant people. Final writeup for this project isn't until January, to me the timing seems pretty perfect to give me something to do when Wikimania is finished :) besides, IEGs are "paid time" whereas Wikimania is purely volunteer time. --EdSaperia (talk) 15:28, 22 April 2014 (UTC)

Flow question

I understand university subscriptions make closed-access journals available to students. That's understandable, as the University has a subscription to the relevant journals.

• Does this proposal involve making existing open-access journals more available, or does it involve also opening access to closed-access journals?
• If the latter, who would pay for the subscription?

--Gryllida 22:46, 19 April 2014 (UTC)

The former–it's about making existing open access journal articles more discoverable. - Lawsonstu (talk) 19:42, 20 April 2014 (UTC)

Aggregated feedback from the committee for Open Access Reader

Moved to grant page EdSaperia (talk) 10:19, 22 May 2014 (UTC)

Round 1 2014 Decision

Congratulations! Your proposal has been selected for an Individual Engagement Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, $6550

Comments regarding this decision:
We appreciate the budget adjustments you’ve made and your willingness to focus scope on first laying the groundwork for the larger vision of this project, and we’re excited to see what you learn from this first phase! Funding for this project will begin in the second half of August, after Wikimania, to ensure we’re not competing for your time :) Please indicate your preferred start date when contacted by our grants administrator (and don’t forget to give yourself a bit of time off for post-mania-recovery - Sept 1 would be perfectly acceptable for a start as well!).

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Questions? Contact us.