Grants:Project/Maxlath/WikidataJS

This project is funded by a Project Grant

proposal

people

timeline & progress

finances

midpoint report

final report

statusselected

Project Grants

WikidataJS

summaryWikidataJS regroups several tools in Javascript to work with Wikidata, designed with quality and developer-friendliness at heart: wikidata-sdk, wikidata-edit, wikidata-cli, wikidata-filter

targetWikidata

type of granttools and software

amount8500 €

type of applicantindividual

grantee• Maxlath

created on13:49, 16 January 2018 (UTC)

Friendly space expectations

Project idea

What is the problem you're trying to solve?

To achieve its mission to give more people more access to more knowledge, the Wikidata community relies heavily on the involvment of software developers, and their ability to turn all sort of ideas to re-use Wikidata data into concrete realizations. Despite counting the largest and fastest growing number of opensource developers, the Javascript community is relatively under-represented and under-served within the Wikimedia movement, mainly because this explosive rise happened within the last 8 years. Part of what makes Javascript special is that it is the only languages web browsers understand, which makes it a highly critical language for any project wishing to see its data being reused by millions of web pages, blossom in all sorts of data visualization, web apps, games... By providing the primitive building blocks of any project needing to read and edit Wikidata programmatically, the WikidataJS tools aim to empower this amazing community of developers, rich of an incredible diversity of members - from UX designers to Data analysts, from hobbyist developers to large organizations technical teams - with the tools it needs to bootstrap and grow their ideas with ease: the lower the barrier to start a project using Wikidata data, the more there will be! Consequently, this grant proposal offers to provide well designed, documented & maintained JS tools to the Wikidata & JS communities.

What is your solution?

Since 2015, and the beginning of my work on Inventaire (an libre/free non-profit project built on top of Wikidata, see #Participants), I encountered many roadblocks on the path to work with Wikidata from either Node.js (server-side) or in web browsers. This led me to start developing the tools that I though were missing:

• wikidata-sdk to query and work with simplified Wikidata data
• wikidata-edit to edit Wikidata from NodeJS

Those are the primitive building blocks any JS developers could install as a dependency to their project to start reading and writing on Wikidata without having to read too much documentation on the Wikidata API specifics, authentification issues, or the Wikidata data model.

An example of how wikidata-cli allows you to fastly access information from Wikidata from the terminal

Additionnaly to those 2 modules that are the real primitive building blocks mentioned earlier, I extended the tool belt with command-line tools:

• wikidata-cli bundles wikidata-sdk and wikidata-edit to read and edit Wikidata from the command-line
• wikidata-filter: a command-line tool to filter a Wikidata dump by claim

While not being specific to JS developers, those tools serve a different purpose: they aim to make working with large datasets from the command-line easier and automatize tasks through shell scripts.

Installation

Those tools are primarily made available to users via NPM, the Node Package Manager. From a terminal, once NPM has been installed, you can install those packages like this:

npm install wikidata-sdk wikidata-edit wikidata-cli wikidata-filter

For people wanting to contribute to those tools development, those tools code is made available on Github, and can be download from a terminal as follow:

git clone https://github.com/maxlath/wikidata-sdk
git clone https://github.com/maxlath/wikidata-edit
git clone https://github.com/maxlath/wikidata-cli
git clone https://github.com/maxlath/wikidata-filter

Those tools are thus neither gadgets, user scripts, nor web apps, but rather a primary brick for any of those: they can be added as dependencies of a webapp, or, for wikidata-sdk and wikidata-edit, fully or partially bundled into a user script or gadget.

Running code documentation

Running code documentation is gather in the wikidata-scripting repository. It can be downloaded from a terminal as follow:

git clone https://github.com/maxlath/wikidata-scripting

You can find there practical uses examples of the tools listed above:

• Add descriptions to Wikidata book entities
• Import external ids from Youtube channel pages into Wikidata
• Import writers pseudonymes from DBpedia to Wikidata
• Load all Wikidata humans items in an ElasticSearch

Other examples of practical uses can be found in the code of things built on top of WikidataJS tools

Some achievements

• wikidata-sdk makes the complex Wikidata data model simpler for newcomers (cf StackOverflow question: Getting readable results from Wikidata)
• wikidata-cli lets you download partial dumps of Wikidata
• WikidataJS modules have accumulated +70k downloads (source: [1] [2] [3] [4])
• wikidata-sdk and wikidata-edit presentation at WikidataCon
• wikidata-cli presentations at WikidataCon
• those tools are now apparently used within Wikimedia [5] [6]

Things built on top of WikidataJS tools

• inventaire.io: the social book inventory built on Wikidata.
  • Uses heavily wikidata-sdk
  • Uses wikidata-edit to let Inventaire users with a Wikidata account edit Wikidata over OAuth
• Hub: a web hub between Wikimedia projects and the wild web
• Synapta is an Italian start-up that used Wikidata-edit in a project involving a custom Wikibase installation and it's also going to use it to add several new Italian monuments and other cultural heritage goods to Wikidata in order to support the Italian edition of WikiLovesMonuments
• citation.js: converts formats like BibTeX, Wikidata JSON and ContentMine JSON to CSL-JSON to convert to other formats like APA, Vancouver and back to BibTeX.
• mapbox-maps: A Google-Maps-like interface for Mapbox maps using Wikidata
• wikidata-scripting: Examples of scripts to edit Wikidata en masse using wikidata-cli
• and more

Project goals

• provide Javascript developers with well designed primitive building blocks to access Wikidata data in an easy, simplified-by-default way
  • this is critical given the place of Javascript as The Web Programming Language

• provide the Wikidata community with quality JS and CLI tools to edit Wikidata, either manually or automatically
  • this is crucial to reach the full potential of an encyclopedia for both humans and bots can edit

• document the tools to help communities around other programming languages to learn from it
  • this effort of documentation is important to maximize the value created by this work, to serve not only the Javascript developers community but all the developers working with Wikidata at large, and potentially help the next generation of tools to emerge after the next paradigm shift, be it in a few years or in several decades

Those goals are the permanently ongoing goals of the project, but, within the context of this grant, should be considered as the project's goals for the 12 coming months: their achievement should be asset at the attested by the project impact metrics.

Project impact

How will you know if you have met your goals?

Different metrics can be used to assess the goals progression (see next section), to which could be added a survey targeting developers using those tools about their experience and satisfaction level.

Do you have any goals around participation or content?

• increase manual editing of Wikidata from the command-line: +10,000 edits
• increase manual editing of Wikidata from third-party applications: +10,000 edits
• increase automated editing of Wikidata from scripts: +1,000,000 edits
• new features & maintenance: +100 commits in total in the various tools repositories
• responsiveness: this grant should allow to address all the issues opened by the community, ideally within a month, and never let more than 5 opened issues/pull requests per repo
• documentation: all features documented in Markdown, at least one code example per feature

Project plan

Activities

• development (code, documentation, answering issues): 95%
• communication (keeping the community updated via social networks and Wikidata status updates, presenting at conferences): 5%

Budget

• development [180h]: 8100 €
  • wikidata-edit and wikidata-cli: introduce a patch-entity operations, to add many claims at once without deleting the existing ones (15h)
  • wikidata-edit: get feature coverage close to 100% (20h)
  • wikidata-cli: support OAuth (20h)
  • wikidata-cli: address security concerns (15h)
  • wikidata-cli: improve inter-operability with other mass data edit tools such as QuickStatements (20h)
  • wikidata-cli: take advantage of the Hub to simplify the wd-open command's code and extend its features (10h)
  • all tools: better support non-Wikidata Wikibase instances (20h)
  • all-tools: process community support requests, new issues and pull requests (40h)

• communication [10h]: 400 €
  • better communicate (blog posts?) on existing but relatively ignored features such as partial/types dumps usind wd-data and wd-sparql commands (3h)
  • take the time to prepare a kick-ass/thorought presentation of those tools for the next Wikicite, or the WikidataCon 2018 (local edition?), that could be re-used in further conferences (5h)
  • communicate all year round on Wikidata channels and social media about the progresses being made (1h)
  • conduct survey on tools impact (1h)

Community engagement

I will keep the Wikidata and JS communities updated on the tools evolutions through social networks (Github, Twitter, Mastodon) and Wikidata internal communication tools (Status updates, External tools list, etc.)

Get involved

Participants

User:Maxlath: I started developing those tools in 2015, at first for the needs of inventaire.io, a libre webapp to do the inventory of your books using Wikidata, and share this inventory with your friends. As the Inventaire project grew, and other people started getting interested in those tools too, I progressively made them grow in completeness and maturity, helped by the community's bug reports, remarks, and code contributions (special thanks to User:JakobVoss!). I have been contributing and advocating for Wikidata since 2013:

• I'm the author of some 260,000 edits on Wikidata mostly on books and their authors, but also on other subjects of interest. A large majority of those edits were made using wikidata-cli based scripts.
• I'm also the author of the Hub
• I have been giving many presentations on Wikidata, both within organized events or informally
• I have been inviting different open source project to consider basing their data model on Wikidata:
  • PeerTube tagging system
  • librehosters software comparison tool

I'm giving all those details on my volunteer work within Wikidata to address concerns expressed by the review committee: this grant proposal isn't a business proposing a service, I'm just a developer 100% dedicated to non-profit and libre software, and would like to continue doing so, despite the fact that I'm currently still not making a living out of it: this grant proposal would hugely help me in sustaining this effort. The rate of 45€/h is a proposition I made considering it was lower than the average freelance developer rate in Western Europe, but I'm of course open to discussions.

Community notification

• Wikidata Project Chat
• Wikidata Status updates
• tweet

Endorsements

Do you think this project should be selected for a Project Grant? Please add your name and rationale for endorsing this project below! (Other constructive feedback is welcome on the discussion page).

•  Support I've tried some of these tools and noticed the great potential, but also some shortcomings that need to be addressed to make them usable by a larger audience. There's been active development in the repositories, which I believe it's worth supporting. --Waldir (talk) 15:08, 29 January 2018 (UTC)
•  Support I have likewise experimented with several of these tools, despite having only a rudimentary knowledge of JavaScript. I agree they have potential for facilitating both editing and consuming Wikidata data. Usability and documentation could do with a boost, which can be expected to drive uptake and then feature suggestions and bug reports. Max has given us lots of examples in the past that suggest he will make good progress with the project. -- Daniel Mietchen (talk) 16:03, 29 January 2018 (UTC)
•  Support Having attended a presentation of Maxlath I am convinced of his commitment to the issue; I also think that the proposal is a valuable one. Papuass (talk) 16:18, 29 January 2018 (UTC)
•  Support This is a valuable resource which will enable more re-use of Wikidata, and it should be funded accordingly. --Magnus Manske (talk) 16:26, 29 January 2018 (UTC)
•  Support We are using wikidata-cli at TransforMap for interacting with our custom Wikibase instance and save a great deal of time by automising data imports with it. Within the last four years of steadily working together, Maxime has never disproven any expectations towards him. This proposal needs complete support, with the suggestion to be more explicit about social aspects being predominant in technical coding practice. See https://twitter.com/izs/status/950077798055661568 --Yalamerde (talk) 21:58, 29 January 2018 (UTC)
•  Support I've contributed to his projects with some pull requests that Maxime adapt quickly to the master code. So I know for sure his ability and I can confirm the benefit of having those libraries maintained. In fact I use this code both for personal, both for working use (cultural heritage projects). --AlessioMela (talk) 23:04, 29 January 2018 (UTC)
•  Support These tools are simply amazing for more advanced users, and for me they have opened up possibilities I didn't even know I had. Jon Harald Søby (talk) 23:28, 29 January 2018 (UTC)
• can be useful to detect and to fix mistakes in wikidata Manu1400 (talk) 23:46, 29 January 2018 (UTC)
•  Support Important project by a maintainer who has been working on it over the last years. Would be excited to see this proceed. --Frimelle (talk) 00:03, 30 January 2018 (UTC)
•  Support I believe we definitely need standard libraries and toolkits for Wikidata. Hjfocs (talk) 09:01, 30 January 2018 (UTC)
•  Support all of these tools are very well done, and their improvement can be used as a foundation to build more reusable components (i'm thinking as webcomponents, react/vuejs components) to allow easier creation of frontend tools to interact with wikidata Atomotic (talk) 11:15, 30 January 2018 (UTC)
•  Support I have been using wikidata-cli since its early announcement and found it immediately useful to explore Wikidata and to make quick demos. Bzg (talk) 12:54, 30 January 2018 (UTC)
•  Support Full support, great tools with big potential. Yarl (talk) 13:50, 30 January 2018 (UTC)
•  Support. I have used the command line interface. This tool is well designed and faithfully maintained. After a sharing that I'd like to be able to create new items with the CLI at WikidataCon, Maxlath implemented the feature before the next day and added it to the present pile. I fully support a grant to fund Maxlath's creative contributions to the ecosystem of Wikidata tools. YULdigitalpreservation (talk) 14:27, 30 January 2018 (UTC)
•  Support I both used and contributed to the project with success. It is maintained excellently and deserves all support! -- JakobVoss (talk) 20:26, 30 January 2018 (UTC)
•  Support We need more developers maintaining reliable libraries to interact with Wikidata and Maxlath can do that very well. − Pintoch (talk) 08:19, 1 February 2018 (UTC)
•  Support I'd like to support this proposal from the Wikidata development team's side. More use of Wikidata's data both inside and outside Wikimedia is crucial in order to really achieve what we built Wikidata for: giving more people more access to more knowledge. More use outside Wikimedia requires good tools like these so others can build meaningful websites on top of our content easily. Maxime has shown a lot of skills in building a project on top of Wikidata's data (inventaire) as well as working with the Wikidata community. His experience as a data re-user is very valuable to make sure what he's building here is actually going to be useful for others. --Lydia Pintscher (WMDE) (talk) 11:31, 16 February 2018 (UTC)
•  Support I used Maxime's tools in a personal project on a custom Wikibase installation and I'm going to use it to add some items on Wikidata: Wikidata-edit was really useful to me and I especially appreciated his responsiveness to the issues I opened on GitHub (issues related with the fact that initially the tool worked well with Wikidata, but not with custom Wikibase instances). --FedericoMorando (talk) 00:25, 02 March 2018 (UTC)
•  Support Slowking4 (talk) 00:06, 12 March 2018 (UTC)
•  Support We must improve the software development in our projects and above all integration with Wikidata --Camelia (talk) 11:12, 29 March 2018 (UTC)