Grants talk:IdeaLab/Getting Academic Reviewers

Untitled

I would go further than simply inviting academic topic experts to contribute. I think we ought to work with universities and department heads to encourage activities such as GA and FA review as a formal part of the goal process.

The performance review of an academic is not simply an assessment of the quality and value of whatever they have accomplished during the last year, it often is a measure of accomplishments versus a specified list of goals. It will be far easier to justify spending time reviewing Wikipedia articles and contributing to GA and FA reviews if those activities are explicitly laid out in the beginning of year goals.

We could be helpful by first gaining an understanding of how such goals are set and in what format they are typically written, and then coming up with draft goals consistent with the style and form of academic goal processes. We would also want to engage with universities and department heads to explain the rationale for this activity and encourage them to ask their academics to consider inclusion of a specific goal along these lines.

Crafting of the goal should be done with care. For example it might include working on several GA nominees but it should never require successful promotion to GA, as that is outside the control of the academic and may lead to serious frustration if there’s a difference of opinion. The good news is that we have a well-established mechanism for tracking activity through edit counts. We might consult with academics to see if other metrics are appropriate and could be easily measured.--Sphilbrick (talk) 18:17, 29 February 2016 (UTC)

I think this is interesting, but I think it is less important. I think of it as a peer review similar to what academics do when they review for journals - I dont think generally they write into their goals to accomplish a certain amount of peer reviews. But it might potentially work anyway, I just think that the more innovative part of the proposal is simply to systematically invite academic specialists directly to provide reviews. The question of how best to motivate them to contribute I think is secondary. Also I think it is important that what we ask them to do is not to contribute text to the article, but to provide an external peer review that wikipedia editors then refer to in the process of editing. Therefore the point about promotion of the article is for me besides the point - all the expert would do would be to point out flaws and opportunities for improvement that editors can then act on. Just as they do in a peer review process. I think the biggest hurdle for acacemics is that they dont understand how the pages are actually edited and they dont have time to the back and forth o the talkpage. If we only solicit peer reviews they will work in a genre they are already comfortable with and will not waste their time in the process of implementing their critiques (just like they dont when they review articles for journals), the editors will take care of that. Maunus (talk) 18:29, 29 February 2016 (UTC)

I don’t see this as an “either – or” but an “and”. I might disagree with a relative importance but that’s not important. I think there’s a lot of merit in your proposal. How to motivate them may be secondary but I think there’s value in thinking through how to motivate them. I’ve heard anecdotally that many academics don’t contribute to Wikipedia and anyway because they get no credit for it. Your proposal provides a convenient and easy way for them to contribute in a meaningful way and if they do so they may managed to get credit for it. I’m “simply” suggesting that we could be helpful in ensuring that they get credit which I think is more important than you realize. But we don’t need to debate which is more important, unless you are concerned that reaching out to the employer’s of academics as opposed to just the academics will dilute resources.

Adding on to your proposal, one of the challenges getting anyone to become an editor is the still high hurdle of our editing experience. One of the beauties of your proposal is that they could work on a peer review and do so in a manner they are quite comfortable, without having to learn anything beyond the most basic markup. We could even make that easier. Why don’t we design a templated GA and FA review form that one could click on and contribute to without having to know a single bit of HTML or media wiki markup? I don’t think that would be hard to do and would make the process not only easier for them but easier for everyone. I know we have some semi standard templates but they could be improved, and design so that they could be used by someone with no markup experience.

Peer review is a process I have personal familiarity with. I spent many years on a peer review committee for a professional organization and several years as the chair of the committee. During that time I not only engaged in hundreds of peer reviews, managed hundreds of others, but also reformed the processes to make it easier for the peer review system to complete their tasks.--Sphilbrick (talk) 19:11, 29 February 2016 (UTC)

I agree, it is an "and" not an "either or", which is why the first thing is to establish a system for contacting academics (or indeed to find out whether this is something the community considers a good idea) - then subsequently one would find ways to increase the motivation of academics to do so (indeed they might not even need much extra motivation). I think the idea of creating a format for contributing reviews of articles would be a really good start. I think the reaching out to employers would take a different and more largescale kind of effort than simply reaching out to academics directly - but it still may be feasible, it would just require more people working and a more centrally organized team of volunteers to do it. I dont have that much experience with peer reviews - having only experienced the process professionally a couple of times on each end of the process - your expertise sounds extremely valuable.Maunus (talk) 21:17, 29 February 2016 (UTC)

I'm managing an expert peer-review here. It was arranged by Lila Tretikov with the editor-in-chief of BMJ (the company that manages The BMJ and many other top-tier medical journals). I've been learning quite a bit from the process. If you'd like me to describe what's going on there, and some things I think the WMF could do to help, I'd be happy to. --Anthonyhcole (talk) 03:02, 1 March 2016 (UTC)

I can see that the procces you are setting up is quite similar to the one I envision, so in that sense you are a couple of steps ahead already. Definitely your experience would be valuable in trying to create a wider effort - perhaps expanding one project at a time rather than going for sitewide use of the process all at once. On your description of your process you mention that "an expert review process would probably involve named reviewers, who would take on accountability for the article" - I agree that experts would have to be named, but I dont think it is reasonable that they assume responsibility for the article - just like a peer reviewer does not assume responsibility for the articles they recommend for publication. It would be enough to have their recommendations on public record I think, then editors must decide how to react to them.--Maunus (talk) 04:09, 1 March 2016 (UTC)

Great, I'll draft some thoughts over the next few days. Regarding your point about responsibility, if the reviewers' names are on the article (at least, on the version they endorsed) they're taking responsibility for its veracity - that's the understanding of the reviewers in that pilot. Still, provided we're agreed on the idea of named reviewers, the motive and consequences don't really matter, since none of that is explicit. --Anthonyhcole (talk) 06:12, 1 March 2016 (UTC)

I dont think we should ask them to endorse any versions, I think just as in an academic peer review, the reviewer simply gives their personal opinion of the article, and then the editors decide how to react to those opinions. I can only see problems stemming from having different versions of the same article endorsed by different experts whose views and perspectives may differ just as much as those of non-expert editors may. Sometimes a reviewer's comments may need to be tempered or even ignored in the light of other concerns (e.g. policy and consensus) or comments from other reviewers.--Maunus (talk) 06:45, 1 March 2016 (UTC)

Anthony’s expert peer-review brings back memories. In addition to experience with peer reviews of professional papers, I once worked on a committee, which spent several years formulating a document attempting to summarize our professions fundamental principles. We used the track changes option in Microsoft word, which produced a document with a lot of similarity to that sandbox. While I am certain we want the final result to be in Media wiki Markup, that doesn’t preclude the concept of an intermediate document, which could be a commercial product such as Word, or perhaps Open Office for those opposed to using commercial software. I am envisioning a process where we could promise that expert reviewers would not have to know any Mark-up, we could copy the print version of the article into a tool, they could do their review almost exactly the same way they would perform a peer review on a journal article, then a volunteer editor would take the output, and make the changes to the article. The hope is that after doing a few, they would be willing to learn how to do it in Markup language.--Sphilbrick (talk) 16:24, 2 March 2016 (UTC)

Really they could just submit it in unformatted text and it could be copy paste or entered directly into a subpage of the article they are reviewing under a generic header. The subpage could then be transcluded where ever we need it. --Maunus (talk) 04:01, 3 March 2016 (UTC)

Initially, I wanted them to do the whole thing on the talk page. I had this little tutorial that I linked them to [1]. But they just sat there and did nothing for a month. Then one copied the article into Word and used the "Review" option to make and track changes and attach comments. Immediately, the others all chimed in and refined the article and added more comments.

The next step was for me to respond to their proposed changes, pointing out where their suggestions went against policy, finding sources to support their proposed changes where I could and asking them for sources when I couldn't, and asking questions to clarify their meaning. But I had to get them on-wiki, and they were frozen in the face of mark-up. In the end, after several false starts, I pretty much replicated the Word document on-wiki (yep, User:Sphilbrick), using our basic table template - something they can easily edit (I hope) using the visual editor.

Yesterday I finished my commentary, and emailed them the link to my sandbox. We'll see how that works. I kept them waiting quite a while, while I stared into middle distance thinking WTF, so I hope I haven't drained every drop of enthusiasm out of them.

I'll paste the review - or a summary - into Talk:Parkinson's disease, once the conversation is finished, and drop a note on en.WikiProject Medicine inviting editors to respond to the review. They know it's coming.

Once the editors and reviewers have agreed on a version, that's the endorsed version, the reliable version. That is, like this version of en.Wikipedia's Dengue fever article, that version of our Parkinson's disease article will be a reliable source, per Wikipedia's WP:RS and WP:MEDRS.

The reviewers were selected by the editorial staff of BMJ, and BMJ overseeing the process. They agreed to recruit the top minds in Parkinson's research for the project and, once this process is complete, the endorsed version of our Parkinson's article will be the most reliable and up to date overview of the topic anywhere. What, if anything, Wikipedia or Wikimedia chooses to do with that endorsed version is up to them.

I underestimated the degree of resistance newbie experts have to wiki-markup. For the next review, I'm definitely looking into something User:Daniel Mietchen pointed me to last year: Hypothesis.is. --Anthonyhcole (talk) 05:20, 3 March 2016 (UTC)

That is a very interesting insight into that process and its challenges. The more I think about it I think that getting experts to contribute article content and getting them to review content is really best tackled as two different processes. Sure one might encourage the other, but I think that simply soliciting external reviews will be much easier than soliciting actual content work in the format you have worked with.--Maunus (talk) 06:36, 3 March 2016 (UTC)

I am in full agreement with this. We want both: contributed article content and expert review of content, but the processes can be different. Contributed content either requires decent knowledge of markup or use of VE, which is coming along but still not quite ready for everything, while expert review can be done either with track changes in a commercial tool or possibly our mockup of something similar. Even if moderately complicated, most expert reviewers will be very familiar with the track changes concept so the learning curve will be minor. My hope is that we would persuade some experts to contribute to the project solely with expert review initially, but then if it is a success — they find it relatively painless to do, and they get credit for it, they might then decide to contribute content. They will still have to deal with a hurdle of markup but by then they may be interested enough in the project to be willing to take that plunge (to mix metaphors).--Sphilbrick (talk) 13:31, 3 March 2016 (UTC)

The WMF has now posted its draft 2016-2017 strategy. I'm concerned they seem to have sidelined support for quality improvement and are emphasising support for just volume and platform improvement. I've left a comment on the discussion page. [2] --Anthonyhcole (talk) 07:31, 6 March 2016 (UTC)

Wikimedia Israel collaborates with HE-Wikipedia editors on getting reviews for featured articles. The editor responsible for the featured articles asks for our help in approaching specialists. We have already done this four times.

We found that specialists who are not from Academia are more willing to help and are as knowledgeable. In our experience academics are hesitant to put their name on a Wikipedia article. For example, for the article about Vipera palaestinae, we approached the chief zoologist of the biggest zoo in Israel to review the article and he gave, among others, an interesting comment about the cultural perspective of the snake in the local Arab speaking communities.

Together with editors we drafted a short letter asking the reviewer to note four main issues:

• The quality of the reference and in particular, if there are additional references in Hebrew
• Clarity - is the article written in a clear style?
• Coverage of the subject - are there missing issues?
• Opening paragraph and wording in general

Michal Lester WMIL לסטר (talk) 16:41, 7 March 2016 (UTC)

It's great to see a chapter directly supporting the language community in improving our reliability like that, Michal. Do you have a staff member specifically assigned to that task? Can you give us a little more detail about how you find the reviewers, please? Do the editors make recommendations? --Anthonyhcole (talk) 09:11, 8 March 2016 (UTC)

The community coordinator is assigned to that task and I’m involved too. The chapter (board, staff and volunteers) has a good network that helps us to get to the “right” specialists. Most of the Hebrew speaking specialists work in Israel, so it is quite easy for us to get in touch with them. Michal לסטר (talk) 16:16, 11 March 2016 (UTC)

It's a marvellous initiative, Michal. I'm so impressed. Wikimedia Israel is clearly committed to reliability. Thank you. --Anthonyhcole (talk) 13:11, 12 March 2016 (UTC)

Thank you Anthonyhcole. Michal לסטר (talk) 13:18, 14 March 2016 (UTC)

Not new idea

This is not a new idea. In Germany we've tried this since - I dunno. 10 years? It sadly did not work. Scientists and Researchers have their own work. They mostly don't have the time and the capacity to do such work. Marcus Cyron (talk) 22:51, 14 March 2016 (UTC)

Scientists contribute their time for free as reviewers of journal articles. The key in this idea is to find a way to exploit that similarity, and to provide motivation to do it. It would be more helpful if you specified the details of how the process worked at de.wiki and what exactly proved to be the problem.--Maunus (talk) 13:16, 16 March 2016 (UTC)

@Marcus Cyron: As is often the case, when someone comes up with a good idea, we often fail to do the research to find out if the idea has been tried before so we can learn from the successes or failures. I’m troubled to hear that this was tried in Germany and failed, but I’d like to hear more information before concluding that it is not worth pursuing. For example, I emphasized above that the effort should not solely be directed at the potential reviewers, but also the people they report to, who help establish the goals of the individuals and ultimately will make the determination whether the time invested is worthwhile. I believe the general view of Wikipedia from a decade ago was at best mixed but largely negative (from the point of view of academics not from the point of view of the average reader). I think this view has changed over time. Wikimedia has witnessed substantial growth in collaborations with universities around the world. To oversimplify, I think the view of Wikipedia from the point of view of universities over the last decade has gone from viewing it as a close to useless annoyance that should best be ignored to a useful tool, which while imperfect has some potential. Thus, I wouldn’t automatically conclude that an experiment done a decade ago means that it would fail today. Can you tell me if the experiment done in Germany included outreach to department chairs and universities as a whole as opposed to potential academics?--Sphilbrick (talk) 18:58, 17 March 2016 (UTC)

• I think that we should to interest the editorial board of scientific journals and the authors of scientific articles to put into the corresponding review articles of Wikipedia their results. For them it will be a great announcement and advertising of their articles, and for Wikipedia replenishment of knowledge. Also it would be nice if the scientific articles in the overview part will refer to Wikipedia. Articles in the Wikipedia should be divided into a brief review page (conservative) and page of the deep knowledge of the material (regularly updated with new data). Dmitry Dzhagarov (talk) 10:48, 15 March 2016 (UTC)

Plagiarized or borrowed with intent to exclude?

The crux of this idea appears to have been plagiarized in large part, or perhaps I should say "borrowed" in large part from Project Accuracy which was created on February 6, 2016, nearly a month prior to this proposal. Project Accuracy had been discussed on the TP of Doc James (moved to the project's discussion page) and was also openly discussed on the TP of Drmies, and is currently being discussed (with mostly negative input from the proposer of this idea) on the TP of Project Accuracy, which actually gave birth to the idea of promoting GAs and FAs to a higher level of reliability via an editorial review board which also would have been peer review because of collaboration with the various project teams whose members will comprise the Editorial Review Board. The Project Accuracy idea was again introduced during the Reach campaign some weeks ago. There is also a grant application for Project Accuracy currently in development. The editor who proposed this academic review idea created a Discussion page for Project Accuracy stating that it would be a mistake to fund Project Accuracy: [3]. Do I smell a COI? I consider such behavior deplorable but I'm not quite sure what action should follow in an effort to find resolution. No one "owns" an idea, but to blatantly lift an ideas presented and published by one editor in order to start your own project and apply for a grant proposal while criticizing the other project is just plain wrong. Atsme^📞📧 18:43, 28 March 2016 (UTC)

Actually I'd say this proposal was closer to this proposal from a few years ago. In particular the idea of having a subpage for the expert review. I doubt even that was the first attempt to propose an expert review system. As for the editorial review board proposal, that seems more like the way Citizendium tried to operate. To me these are two very different proposals. Getting Academic reviewers and doing so in a way that works for both us and them is an interesting challenge, and I think this proposal is pretty much there. But the key ingredient is a culture change in Academia to give credit for such reviews. That will eventually come, but we could be years or even decades from it being mainstream. Wikiproject Accuracy is more a revival of the idea of shifting Wikipedia to a Citizendium model. This has an even longer gestation period, and Citizendium itself an illustration of the problems in that approach. WereSpielChequers (talk) 19:24, 28 March 2016 (UTC)

I struck my comments and admit they were instigated by the discussion to not fund Project Accuracy. I will support whatever works best for the project, whether it turns out to be this endeavor or Project Accuracy. It is something we should all be working together to achieve. I apologize for overreacting, and hope we can move on to bigger and better things. Atsme^📞📧 19:37, 28 March 2016 (UTC)

Atsme (talk · contribs)I do think we should think about joining forces for the shared goal of improving the quality of content. I think it is logical to start with using existing project infrastructure and processes, and to work with unpaid volunteer labor as long as possible. I think that in that way potentially my project could be a step on the way towards yours. I think yours is much more ambitious in its scope and much more at risk of facing serious community resistance. I think this risk exists to a degree in my idea, but I have purposely tried to minimize the risk factors of ringing the warning bells for "COI" "paid editing" and "cabals" all of which have proved to be major redflags for the wikipedia community in the past. My main concern with your project is that you ring all of those bells and do not seem to be aware of it.--Maunus (talk) 18:35, 29 March 2016 (UTC)

And for the record I had not seen your project or the previous idea about expert review when I had mine. My idea emerges from an article that I wrote urging experts on anthropology and native american subjects to participate in writing wikipedia. The article was submitted last summer and will be published this fall in American Anthropologist. --Maunus (talk) 18:37, 29 March 2016 (UTC)

Thanks. WereSpielChequers (talk) 20:05, 28 March 2016 (UTC)

Thank you for clarifying. I looked at the one WSC linked to, and saw where it was suggested to merge it with a slew of others I have not read. Which one of those proposals actually sprouted wings? What it tells me is that we have an ongoing problem that isn't being resolved, so let's join forces and make it all happen in a good way. I have a bit more disposable time on my hands now that I'm retired from a very long and time consuming career in the television industry, although I'm still involved in ranching and probably will be until my last breath. Maunus, if you WereSpielChequers, Northamerica1000, and I formally announce that we're Project Coordinators for Project Accuracy, we'll only need one more editor to comprise the 5 primary coordinators. I've invited Doc James to join as a PC, or at least take the lead in helping us form the review board, or committee or team. We can make that decision once we've formalized who our PCs are, so what say ye? Yay, or nay? Atsme^📞📧 19:21, 29 March 2016 (UTC)

I would need to know first how the project would answer the questions I posted at the project talkpage. I think the risk of community backlash to anything that looks like it is creating a content "cabal" is the greatest - so I think the questions about how the project will work with the existing consensus based review processes is going to work. I think it would be much easier to start out with the peer review project (which already has a lot of people ready to volunteer), since it doesnt require making any institutional changes to existing processes, and then if that works out move towards institutionalizing it as a wikiproject. Basically the project could start simply with a group of reviewers who decide to participate in all FA reviews by submitting accuracy checks and solicit peer reviews from academic topic experts. This would require no more coordination than what could be done on a user talkpage. Also why start with a hierarchical structure with formal PCs? I would be happy to participate in a brain storming session (on or off wiki) on how to create a process that combines aspects of what we have both proposed, but I could not commit to being a coordinator of anything before I know what it is.--Maunus (talk) 19:40, 29 March 2016 (UTC)

Dear Atsme. Thanks for the offer, I'm very flattered to be offered such a post. However I'm not sure I would be worthy to be a Project Coordinator of a project seeking expert review from academia. Also there are ways to do this with community support, and ways to do it that would pretty much guarantee conflict. I don't want to be involved in something that hasn't learned from past failures. Like Maunus I have big concerns as to how your proposal would work, even if you drop the hierarchy and rename it to something less likely to rile people. One point you might want to consider is that a big move towards getting expert review would need to start with pilot projects. WereSpielChequers (talk) 21:51, 29 March 2016 (UTC)

Some ideas

So here are some ideas on how to systematize this and maximize its effects:

• Crowdsource a list of relevant academics to contact on the article's talk page. This list should be a template and found somewhere in the article's header in collapsed state. This is the template which can be used to control and optimize the academic-recruitment-process.
  • Crowdsourcing the list yields far more and better results than personal endeavors.
  • Instead of finding someone to contact the academics per article etc. there could be a dedicated taskforce / WikiProject. So for instance there could be a way to let the academics-list-template make note of itself once it reaches a specified count of entries or age (e.g. by adding a category such as "Category:Articles with lists of academics to contact older than 5 months" to the talk page) with the academics-outreach-taskforce going over each of these if they haven't been carried out so far by the people of the article's talk page. They could form the interface of Wikipedia to the academic world as the moderators of reddit's /r/IAMA are between reddit and the celebrity-world. They could investigate the most efficient ways for such recruitment and optimize their processes, texts etc. They could also help out academics with getting along with the site and answer their questions. They could also help out people on talk pages who organize the recruitment (typically just sending a few emails once the list is done) on their own. A typical job for members of this taskforce would be to check the above mentioned category, google the contact information of each member on the list, use the most appropriate email-template for the academics of the list (and/or the article's topic), adjust it a bit, send the mails, and edit the list-template on the talk page so people know who was contacted already. Generally it's left to the people of the article's talk page to contact the academics. The taskforce just takes over the ones they don't want to or don't feel competent enough to contact.
  • The people of the article's talk page can also crowdsource a list of things they want the academics to know - such as current controversies among the article's editors, what information is missing, what they'd like the academics to do most, etc. The previously mentioned taskforce would also have to adjust their mails according to this.
  • This isn't just about academic reviewers but also about academic editors.
  • Instead of having to use Wikipedia's editor there could be an add button in the template which allows people (including newcomers) to easily add people to the list. This requires some software changes though.
  • (Note) It might not be a good idea to include contact information right in the list - spammers etc. might make use of this, so it might be a good idea to prohibit people from posting their email addresses or alike. One could possibly also allow the posting of their mail adresses but have it hidden from the public and just accessible via a specific process or by specific members of the above mentioned taskforce.
  • For each academic there probably also needs to be a reason field or alike in which one is typically expected to post a paper, news-article, or rationale for why the academic should be recruited for the specified topic.

• There could be some kind of official award for academics who review Wikipedia articles (for specified numbers of reviews; one-sentence and troll-reviews are of course omitted). They can embed these on their websites or print them out to show that they're doing this public service.

• Instead of just asking individual universities and academic networks there could be some kind of software - a Wikipedia Reviewer - which can be downloaded by ^{(in the case that it's not a web-app)} and distributed to the universities etc. The software could introduce academics into the most basic Wikipedia syntax / editing, take over the registration and (for academics) identity-confirmation process, have properly usable UI for the purpose of reviewing etc.

• There could be an Academic Review Day which has a banner on Wikipedia asking academics to review and contribute.

--Fixuture (talk) 22:17, 29 March 2016 (UTC)

Related problem: Reduced Wikipedias value as a source of new scientific information

All of Wikipedia content is meant to be supported by secondary sources, as per en:WP:RS. Anything biomedical is also subject to en:WP:MEDRS which is even stronger in that. Wikipedia:Identifying_reliable_sources_(medicine)#Respect_secondary_sources says: "Scientific findings are often touted in the popular press as soon as the original, primary research report is released, and before the scientific community has had an opportunity to analyze the new results. Such sources should generally be entirely omitted (in accordance with recentism), because determining the weight to give to such a study requires reliable secondary sources (not press releases or newspaper articles based on them)." However, reviews appear with a delay of six months - a year. On some topics overview never will be written. For students and young researchers the only source of information about new research is an article in Science Daily or short review in Nature. But they usually do not show the place and role of this concrete investigation in the context of the general direction of this science. Wikipedia can help in this. Dmitry Dzhagarov (talk) 09:57, 15 March 2016 (UTC)

I dont think this is really a closely related problem. In fact I dont think it is a problem at all. Encyclopedias should not be about getting the most recent knowledge, but only about getting the best established knowledge. We should not be trying to crystalball which research results will stand the test of time and which will not. That is what academics are for.--Maunus (talk) 00:30, 30 March 2016 (UTC)

Time to get to work?

I think there is some critical momentum here to start working on this idea. I think the easiest way to proceed is for everyone who wants to participate to simply select articles for which they will take the responsibility of approaching potential peer academic peer reviewers. I will be happy to start with articles that are within my area of interest and for which I think I will be able to identify the right people. I think the first steps for us should be.

1. Create a centralized place on en.wiki where coordinating discussions can take place. I suggest en:Wikipedia:Peer review/External_Peer_Reviews.
2. Write a standardized letter of invitation that describes the task (the type of review solicited, a rubric of areas to be evaluated, the way to sumit the review (e-mail?) etc.), describes the option of being publicly acknowledged as an academic peer reviewer on wikipedia, and any other reasonable offers of quid-pro-quo we can think of (maybe an official letter of recognition for their work on official stationary that they can add to their tenure file). I think the best way to make sure they know what to do is to turn the current version of the article to be reviewed into a pdf file that they can then either comment on directly in adobe or comment on separately in a text format. We can then transfer the comments to the article's peer review subpage.

And that is pretty much it. I think I will begin within the next couple of weeks. But lets discuss in this section if there are other aspects we can start working on right away as well, and what the next steps should be.--Maunus (talk) 00:12, 30 March 2016 (UTC)

• @Maunus: JFYI: in my suggestion further up in #Some ideas the centralized place of coordination would be a WikiProject / taskforce which would also be concerned with this standardized letter of invitation. Also supporting an "official letter of recognition for their work on official stationary that they can add to their tenure file" or alike (I suggested an official award above). I would find it great if you'd get this going! --Fixuture (talk) 17:54, 30 March 2016 (UTC)

I am not sure I see a need currently to establish a separate wikiproject, we already have WP:PEERREVIEW, which fits the bill well. I think potentially making a separate project should wait untill we see if there is sufficient traction and community support for this idea. Let's start by organizing at the subpage of WP:Peer review I linked above. Really all we need at this point is a place to talk.--Maunus (talk) 18:30, 30 March 2016 (UTC)

Funding needs?

Hi Maunus. I'm a community organizer for the WMF who initiated this campaign on content curation & review, and I also work closely with the WMF grant programs. Thanks for taking the time to write up this proposal on inviting academic experts to provide reviews of article content. I know a number of ideas have been pitched here on the talk page and on the project page about how best to achieve this; I wanted to ask what role you see yourself having in this project seeing as there is clearly some community interest, and if you're considering a grant request to support this project. For instance, some activities that have been mentioned:

• building new templates or pages that would facilitate some form of recognition that's been discussed,
• conducting extensive outreach efforts to academic communities (e.g. e-mails, but beyond that, perhaps going to a conference, visiting universities)
• general project management time for matters of scheduling, coordinating volunteer efforts, follow-up with academic experts after initial invitations, etc.

Of course, WMF grants can't directly fund content creation or reviews, but proposals that facilitate those kinds of contributions from volunteers are certainly eligible, and that's more or less how this project reads to me. With that said, if you don't feel you need any particular funding, that's fine too. But I wanted to gauge whether it could be helpful in this project. Let me know what you think. I JethroBT (WMF) (talk) 08:55, 31 March 2016 (UTC)

Thanks, I JethroBT (WMF) - Personally, I think I would want to exhaust the possibilities that do not require funding first. That would give a better idea of what is possible and what can be achieved better with allocated funds. I have a personal preference for loose organization and flexibility, so accounting for grant funds and budgeting is not something I would find particularly appealing. However, if someone else thinks of ways that this project could be supported by grant funds, they are more than welcome to do the paper work. I don't feel any particular ownership of the idea, and anyone is welcome to take over the reins on any aspect of the project if they have the energy and inspiration to do so.--Maunus (talk) 20:32, 31 March 2016 (UTC)

• In passing, I note that Of course, WMF grants can't directly fund content creation or reviews. Why not?

April 12 Proposal Deadline: Is your project ready for funding?

The deadline for Individual Engagement Grant (IEG) submissions this round is April 12th, 2016. If you’ve developed your idea into a project that would benefit from funding, consider applying!

To apply, you must (1) create a draft request using the “Expand into an Individual Engagement Grant” button on your idea page, (2) complete the proposal entirely, filling in all empty fields, and (3) change the status from "draft" to "proposed." As soon as you’re ready, you should begin to invite any communities affected by your project to provide feedback on your proposal talk page.

If you have any questions about IEG or would like support in developing your proposal, we're hosting a few proposal help sessions this month in Google Hangouts:

• April 5th, 16:00 - 17:00 UTC
• April 8th 1:00 - 2:00 UTC
• April 12th 16:00 - 17:00 UTC

I'm also happy to set up an individual session. With thanks, I JethroBT (WMF) 00:38, 2 April 2016 (UTC)

My own experience with expert reviews

For a few months now, I have been involved in the Primary School Research project. As part of the project, we contact external experts and ask them to review certain articles. I would like to comment on the process.

1. we first contact them to ask their help in reviewing an article. This is done by email. In the email, we explicitely mention that the review will be published under a free licence (cc by sa 3.0) on wikipedia
2. upon their consent, we extract a pdf version of the article and send them by email with a form to fill up. This form again explicitely mention that the review is published under a free licence, with a check box
3. the expert send back the review in pdf format, attached to an email
4. I publish the reviews on commons ([Research:Wikipedia Primary School SSAJRP programme/Review List|here]]

At this stage... everything is generally fine. Things get hot when it come to recording the license status of these reviews. I usually forward to permission the email sent by the reviewer with the review attached, and I specify the url where the document may be found. I met several cases:

1. sometimes, we are lucky and the permission is recorded. The pdf is taggued OTRS approved and we are good
2. sometimes, we are less lucky and we are informed that the mention of the free licence on the document is not sufficient, and that the reviewer should fill up a consent template and send it back by email, either to us or to permission directly. Even though the reviewer already agreed to the free licence on the pdf, we have to contact him again to ask him to repeat his consent explicitely. The challenge at this point is that the reviewer sometimes only send the permission template to us. We then forward it to OTRS (which some OTRS agents consider is ok).
3. in some cases, this is sufficient. The pdf gets taggued OTRS approved and we are good.
4. in other cases, the forwarded template consent is rejected because "forwarded consents" are not acceptable. We are asked to go back a 3rd time to the expert to ask him to AGAIN send the consent, but to permissions mandatorily. At this point, we better have a very understanding expert
5. usually... due to delays... at this point... the review has also been deleted anyway (eg [4])

Bottom line... our approval process at OTRS is simply not very consistent. I would not blame agents... I am one myself. But the thing is that the mandatory process is not consistent and this should be taken into account for any "Expert Review Process Set Up". The funny thing is that this is not to publish something that is likely to be reused anywhere anyway :)
So since this is a research project and we must test options... I have tried another option, which is to directly copy and paste the review elements in the talk page of the article (example). At this stage... no one has ever deleted the reviews copied in the wiki.

I think that... we have a special challenge here. On one hand, we should not expect experts to come edit their comments and their reviews directly on the wiki. Some people actually do it, but it is too much asking in most cases.
But on the other hand... getting the reviews uploaded on Commons and approved by OTRS is adding a layer of bureaucracy that is not so cool to request from reviewers. I guess we need to find a middle option that could work well. Anthere (talk) 21:06, 6 April 2016 (UTC)

Wow, that does sound like bureaucracy is puttin some hurdles for this kind of thing. Presumably this could be avoided simply by having the reviewer copypaste the review text unformatted into a wikipage. No?--Maunus (talk) 01:48, 7 April 2016 (UTC)

Yes and no. If he were doing it without an account created, some might raise the point (valid point) that the review can not be given as "review from guy X" since there is no proof that the uploader is guy X. So whilst it would fix the licence issue, it would damage the "expert approach". If we ask the person to actually create an account, I guess he would potentially be requested to prove his identity (at a minimum with an email to OTRS ?). So still bureaucracy. Anthere (talk)

I think that would be a fairly silly point of doubt to raise actually, if we know that a review has been requested from a specific person. AGF and common sense applies. IPs cannot be assumed to be liars. Especially not when there is a very good reason to assume they are who they say they are. --Maunus (talk) 23:53, 11 April 2016 (UTC)

Peer Review Reform

I am including aspects of this proposal in a proposal for peer review reform that I am developing at the English language Wikipedia at En:User talk:Maunus/PeerReviewReform. Please feel welcome to come and give your input to the proposal as I am developing it.--Maunus (talk) 08:58, 11 August 2016 (UTC)