Jump to content

Knowledge Engine

From Meta, a Wikimedia project coordination wiki

"Knowledge Engine" is a term used by the Wikimedia Foundation in 2015 for an evolving set of ideas. The term appears to have been first used in April 2015; there was some effort to leave the name behind in October (?) 2015, in favor of the name Discovery, but by that time it had already been included as the title of the project in a grant from the Knight Foundation. Knowledge Engine became a central component in what many have termed a "leadership crisis" at the Wikimedia Foundation in early 2016.

Knowledge Engine has been covered in the following places:

Knowledge Engine by Wikipedia will democratize the discovery of media, news and information -- it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. ... Knowledge Engine by Wikipedia will be the Internet's first transparent search engine.

From the Knowledge Engine grant proposal

Note: The original contents of this page (in the shaded box below) were posted to User talk:LilaTretikov (WMF), the user talk page of the Wikimedia Foundation's executive director. It was initially posted anonymously, but Lila later signed it. I have moved it here, and I would suggest discussion should continue on the talk page, and further information about the Knowledge Engine could be added to this page outside the purple box. -Pete F (talk) 19:55, 30 January 2016 (UTC)[reply]
edit  

Knowledge Engine grant

Some background on the Knowledge Engine grant

Hi Everyone,

As some of you know, I have been making concerted efforts to engage deeper on-wikis and to provide more insights into my thought process. As a demonstration of this commitment, I would like to share my thoughts on the Knight Foundation grant which has been called out for clarification.

What are the new WMF initiatives which this grant supported?

The text (after the bullet points) below is from the actual grant paperwork, and is duplicated a bit further down. Here I’d just like to highlight the functional areas of our WMF strategy which these initiatives touch.

Reach:

  • Test results from search and user testing
  • An improved search engine and API for Wikipedia searches
  • Measure user satisfaction (by analyzing rate at which queries surface relevant content)
  • Can the Wikimedia Foundation get Wikipedia embedded via carriers and Original Equipment Manufacturers?

Community:

  • A public-facing dashboard of core metrics used in product development
  • A sample prototype on a small dataset to showcase possibilities
  • Create a public-facing dashboard of key KPIs
  • Use Key Performance (KPIs) to inform product iteration, and establish key understanding and feature development for the prototypes
  • Measure application Programming Interface (API) usage

Knowledge:

  • Test results exploring relevance of content surfaced
  • Measure no results rate

Knowledge + Reach:

  • Would users go to Wikipedia if it were an open channel beyond an encyclopedia?

Reach + Community:

  • Conduct tests with potential users
  • Measure user-perceived load time

Why didn’t you discuss these ideas with the community sooner?

It was my mistake to not initiate this ideation on-wiki. Quite honestly, I really wish I could start this discussion over in a more collaborative way, knowing what I know today. Of course, that’s retrospecting with a firmer understanding of what the ideas are, and what is worthy of actually discussing. In the staff June Metrics meeting in 2015, the ideation was beginning to form in my mind from what I was learning through various conversations with staff. I had begun visualizing open knowledge existing in the shape of a universe. I saw the Wikimedia movement as the most motivated and sincere group of beings, united in their mission to build a rocket to explore Universal Free Knowledge. The words “search” and “discovery” and “knowledge” swam around in my mind with some rocket to navigate it. However, “rocket” didn’t seem to work, but in my mind, the rocket was really just an engine, or a portal, a TARDIS, that transports people on their journey through Universal Free Knowledge.

From the perspective I had in June, however, I was unprepared for the impact uttering the words “Knowledge Engine” would have. Can we all just take a moment and mercifully admit: it’s a catchy name. Perhaps not a great one or entirely appropriate in our context (hence we don’t use it any more). I was motivated. I didn’t yet know exactly what we needed to build, or how we would end up building it. I could’ve really used your insight and guidance to help shape the ideas, and model the improvements, and test and verify the impacts.

However, I was too afraid of engaging the community early on.

Why do you think that was?

I have a few thoughts, and would like to share them with you separately, as a wider topic. Either way, this was a mistake I have learned enormously from.

Was the $250,000 Knowledge Engine grant a restricted grant?

Yes. Let's talk about restricted grants and the WMF. The Foundation has taken restricted grants in the past, per our policy, especially when we were a much younger organization. As our most recent audit report shows, we have received restricted grants more recently for Visual Editor and Wikipedia Zero and Mobile (in 2014-15). With this grant we brought the idea to the funder and they supported our work with this grant. To be clear, this is not an instance of a funder driving WMF's agenda. They provided financial support to the plans we presented to them.

I’d like to take this moment to call out the requirement for Board of Trustees approval for 1) grants over $100,000 2) grants which do not conform to Foundation policies, and 3) grants which create financially instability for the Foundation. The Knowledge Engine grant was unanimously approved by the WMF Board of Trustees. All members voted and approved (see minutes) the grant on November 7th. The motion was made by James, and seconded by Denny.

Why did the WMF Board of Trustees vote to accept this grant?

While I cannot speak on behalf of the board, I can share my take: Restricted grants not only benefit budding organizations, but they also aide existing organizations with new initiatives. In the early stages of discovering what would eventually become “Discovery”, we decided to apply for this grant. Our aim was to begin exploring new initiatives that could help address the challenges that Wikipedia is facing, especially as other sources and methods arise for people to acquire knowledge. If you haven’t yet, please have a look at the recent data and metrics which illustrate the downward trajectory our movement faces with readership decline (since 2013), editor decline (since 2007, which we stabilized for English Wikipedia in 2015), and our long standing struggle with conversion from reading to editing. These risks rank very high on my list of priorities, because they threaten the very core of our mission. The time for investigating new approaches at the foundation to address these risks is absolutely now. Thus, the Board approved this grant to help us fund investigating and developing new ways of reaching and serving our readers.

Why should the community and staff support this decision of our board and leadership?

I would hope that for staff, the answer to this question is clear.

That brings this to a discussion more centered around, why should the community support the decisions of the WMF board and leadership? Well the honest truth is, the community has no obligation at all to support the WMF board. It is very much the other way around, the Foundation is accountable to our readers, contributors, and donors. The Board and the Foundation both act, in their own capacities and to the best of their limited abilities, to further the mission of the movement. Sometimes we get it wrong, sometimes we get it right. But it is entirely up to each individual contributor to decide whether they want to support a given initiative or not.

I’d like to have an open discussion, on a later date, on how values, policies and duty tie into interpersonal relationships and transparency, organizational productivity, technological innovation and long term relevancy of our movement and our projets.

Some reading refresh I did for this discussion:

Why did the board not publish this grant paperwork?

Generally we do not post donor documents without advance agreement, because doing so breaks donor privacy required in maintaining sustainable donor relations. In practice, I am told we have not actually published grant paperwork since 2010, but we do publicize grants in blogs when requested and agreed to by donors. A portion of the KF Knowledge Engine grant document that outlines the actual commitments we’ve made I quoted below.

What do we want to investigate that was covered by this grant?

We hypothesize that if we help our users discover more Wikimedia content through search, more users will come to us and perhaps more will engage in editing. Our basic search brings zero results roughly 30% of the time - a problem which, once fixed, extends our reach. We also want to learn if exposing sister-project content and other open sources (like open street maps) through our search interface will help our readers find and read more of our content. At the time we called this concept “Knowledge Engine”. Today, we call this “Discovery” because that is the phase the team is in. The Discovery team is actively working with Wikidata, open maps and APIs; and you can read more about it on the Discovery FAQ page.

What are the expected outcomes of this grant? (quoted text from the grant)

At the conclusion of the first stage, the results will include:

Test results exploring relevance of content surfaced

Test results from search and user testing
An improved search engine and API for Wikipedia searches
A public-facing dashboard of core metrics used in product development
A sample prototype on a small dataset to showcase possibilities

What are the activities this grant supports? (quoted text from the grant)

Answer key questions:

Would users go to Wikipedia if it were an open channel beyond an encyclopedia?
Can the Wikimedia Foundation get Wikipedia embedded via carriers and Original Equipment Manufacturers?
Use Key Performance (KPIs) to inform product iteration, and establish key understanding and feature development for the prototypes
Conduct tests with potential users
Create a public-facing dashboard of key KPIs
Measure:
User satisfaction (by analyzing rate at which queries surface relevant content)
User-perceived load time
No results rate
Application Programming Interface (API) usage

What is an example of discoveries we made so far?

Portal First A/B/C test report (2 tests ran at the same time) .

With the first test group, we did not find reliable improvement as the delta was between -0.9% and 2.8%. Since it is not consistently positive, it is not reliable. With the second test group, 1.7% to 5.5% more sessions were likely to end in a clickthrough compared to the control. This represents between 300 and 1.3 million more people every day likely to go through to read an article our editors wrote. These people “bounced” from our site before. This is a tangible improvement, and I’d like to thank Discovery for the great work they have been doing.

Please help us shape these ideas and validate them.

Please help provide insight on Discovery team work on the product portal pages. You can read and comment on our tests and help submit ideas in Phabricator. I am confident you can help us with both observations, opinions, ideas and safeguards. Ultimately, we’ve just started the Discovery process, and I’m hoping we can give it a clean start when it comes to ideating together, please don’t bite the newbie. I assume good faith, and I hope you do too. Let’s all treat each other with civility and etiquette, and see if we can collaborate to build a consensus on the WMF’s project direction to help readers discover the high quality content and knowledge our editors are creating.

A fellow humble child of knowledge, LilaTretikov (WMF) (talk) 22:46, 29 January 2016 (UTC)[reply]


Thanks Lila, that's interesting context. I believe the phrase "knowledge engine" has been used to describe WP or aspects of WM since at least 2014, when it showed up in the annual report. I've heard it used with various other connotations since then, but this seems as good a description as any of a constellation of tools, living reference works, and communications channels devoted to synthesizing and organizing information into usable knowledge. –SJ talk  06:01, 30 January 2016 (UTC)[reply]

Questions

[edit]
  • I have a question which I do not see covered above: whose employment is dependent on this restricted grant?
  • I work on that team. The work you report as supported by this grant, is my work. I wrote the dashboards, my team performs the A/B tests. I had no idea that there was a restricted grant covering this work until now. And quite honestly it scares me quite a bit to hear that a one-off donation is the thing resourcing my work - because I feel like my work is valuable to the movement's progress - and my employment, because I like making rent. So that's my question; what specifically is this being spent on? Not, what are the goals of the project, but whose existence is supported by it? And what is going to happen when that grant expires? Ironholds (talk) 15:15, 30 January 2016 (UTC)[reply]

I simply don't understand what is being said here. Lila, you say that the Wikipedia search engine should be improved. Alright. But what's that got to do with a "vision" of "free knowledge as a universe"? I don't understand even the mere meaning of this phrase. I don't understand what the question means whether "the Wikimedia Foundation can get Wikipedia embedded via carriers and Original Equipment Manufacturers". What does embedding mean here, what do you mean by carriers, which OEMs? You are asking: "Would users go to Wikipedia if it were an open channel beyond an encyclopedia?" What's that got to mean? What is "an open channel beyond an encyclopedia"? It all sounds very high-profile, not at all like the rather understandable task of improving a search engine. Isn't it possible to put in clear words what this Knight grant is good for?--Mautpreller (talk) 19:16, 30 January 2016 (UTC)[reply]

Mautpreller, I agree, it is difficult to discern a clear narrative here, though LilaTretikov (WMF) has made several things clear. The paragraph about "free knowledge as a universe" strikes me as the kind of thing one would expect to hear from an artist, or maybe from a company leader for an organization heavily rooted in that person's personality (e.g. Richard Branson). Deep introspection about the nature of knowledge and future trends is worthwhile, but for an organization and movement like Wikimedia, it's out of step with how we operate.
In my view, the 2010 strategic planning process, and many of the things that flowed out of it (e.g., the grant programs), constituted a substantial investment in making things clear and predictable, so that the various creative forces throughout our global movement could plan around a stable central corporate entity. That of course was not the case in software development, but many WMF efforts carried this kind of thinking forward; the rewrite of the Terms of Use, and other legal documents, comes to mind.
Based on what's written above, I'm worried that Lila has a fundamentally flawed understanding of what is desirable or possible from her position. She characterizes her failure to communicate as a "mistake," but I'm not sure she recognizes that it's a mistake in the very core function of her job. We simply cannot function with an executive director whose personal fear of interaction drives her activities like this. At least, not optimally -- not by a long shot. -Pete F (talk) 01:07, 31 January 2016 (UTC)[reply]

What is being requested

[edit]

What has been requested is the "grant application". This is a document prepared by the WMF and submitted to the Knight Foundation rather than a document from the donor.

All other moment entities, including chapters and those applying for individual engagement grants publicly post proposals for funding. I do not understanding the reasons the WMF cannot also? Doc James (talk · contribs · email) 16:56, 30 January 2016 (UTC)[reply]

I would like to see a citation for the claim that all other movement entities publicly post proposals for funding. In particular, I would be rather surprised if all of the EU research project submissions to FP6, FP7, and Horizon2020 are being published for the chapters that participate in them. Do you have any support for this statement, James? --denny (talk) 17:39, 30 January 2016 (UTC)[reply]
I think that is certainly not the case, as you say, Denny. My guess is Doc James meant "proposals to WMF for funding". Those have indeed been publicly documented every time (with only one exception I'm aware of in my tenure (since 2011)). Asaf (WMF) (talk) 22:27, 30 January 2016 (UTC)[reply]
James, care to clarify what you meant? Since Asaf and Pete seem to think you meant something else than what you wrote. --denny (talk) 22:44, 30 January 2016 (UTC)[reply]
Denny, I would suggest that this page is not a great venue for settling personal differences. It is very clear that Doc James' statement was not entirely accurate; if you feel that inaccuracy is something worth fully litigating, could you please do so at his user talk page, or somewhere else, so that we can focus on the Knowledge Engine here? -Pete F (talk) 05:35, 31 January 2016 (UTC)[reply]

Lila, Sj, Ironholds, Doc James, Denny: Having a document like this published on a user talk page seems sub-optimal. Those who are interested in this topic (as opposed to those interested in this user's thoughts generally) can't watchlist it here; among other issues. Any problem if I move this elsewhere for continued discussion -- e.g., Lila Tretikov's remarks on the Knowledge Engine, January 2016? -Pete F (talk) 17:56, 30 January 2016 (UTC)[reply]

Or better yet, simply Knowledge Engine -- with a section reserved for Lila's comments. -Pete F (talk) 17:59, 30 January 2016 (UTC)[reply]
I've now moved the text. Comments or other solutions welcome. -Pete F (talk) 19:43, 30 January 2016 (UTC)[reply]

Denny and Doc James, While I am not intimately familiar with the FDC/Annual Plan grant program, the WMF's other grants programs are a stellar example of participatory grantmaking (as described in depth in report commissioned by Asaf (WMF) and Katy Love, I believe). I don't know whether Doc James' statement is literally accurate or not, but it clearly references the values deeply woven into Wikimedia's approach to grant-giving. -Pete F (talk) 21:08, 30 January 2016 (UTC)[reply]

For the record, I was not involved with commissioning that report. I was, however, interviewed at some length by its author. Asaf (WMF) (talk) 22:16, 30 January 2016 (UTC)[reply]

Contract published

[edit]
Yes that is a really interesting document as it gives a fairly clear description of what we are doing with passages such as "Knowledge Engine by Wikipedia will be the Internet's first transparent search engine, and the first one originated by the Wikimedia Foundation". It however goes beyond what Lila says above and makes clear that we are building "a system for discovering reliable and trustworthy public information on the Internet" not just planning to improve internal search. Doc James (talk · contribs · email) 23:08, 11 February 2016 (UTC)[reply]

FAQ for the Knowledge Engine

[edit]

Is here Knowledge Engine/FAQ Doc James (talk · contribs · email) 03:06, 19 February 2016 (UTC)[reply]

Retrieved from "https://meta.wikimedia.org/w/index.php?title=Knowledge_Engine&oldid=26861901"