Grants talk:IEG/Wikiscan multi-wiki/Midpoint

Data Lake

This is just an FYI that has to do with the hardest part of the project which is scaling for 300 wikis. The analytics team is working on an analytics-friendly data store on labs that you will be able to query to harvest your data, unlike the labs replicas this database will have an analytics-friendly schema: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/Mediawiki_page_history

This data is now available privately and we are vetting it, it will power the new Wikistats site, here ais our request for feedback where you can leave comments, we will really appreciate those as you know the space: https://www.mediawiki.org/wiki/Wikistats_2.0_Design_Project/RequestforFeedback/Round2 NRuiz (WMF) (talk) 20:46, 18 April 2017 (UTC)

Hi NRuiz (WMF), thank you for your message, I will check this new datasource when it is available. I see several good reasons to use it instead of the raw database, especially at the edits level [1], but I'll have to see pros and cons, for example if the edits history is not available for the current day. It will probably don't help much for scaling issues because they are mainly on the server for local stats calculations. Akeron (talk) 15:01, 9 June 2017 (UTC)

Report accepted

Dear Akeron,

Thank you for submitting this Midpoint Report. I am accepting it now, with the following comments:

• Congratulations on released the beta version of Wikiscan! It was fun to read the comments you linked to from the French community. Bravo, indeed.  :-)
• In your Final Report, please plan to describe your methods and activities in more detail than you have in this Midpoint Report. The purpose is both to give others an opportunity to provide feedback on your methods and activities, and to allow them to learn from you. Many IEGrantees work alone, and it is still possible for them to provide an overview of their methods and activities.
• In regard to your comment that you wish you'd grouped similar tasks together, please let me know if you'd like to make any revisions to your current plan now. We can discuss this.
• You mentioned that you may consider applying for more funding in the future. You may already be aware that the IEG program is now closed, so you would apply for future funds through Project Grants. Further developments of this project will need to take into consideration any related work being developed by the Wikimedia Foundation or other volunteers, so keep that in mind as you think about moving forward.

Congratulations again on the work to date. Our Grants Administrator will be in touch with you about your next disbursement.

--Marti (WMF) (talk) 22:47, 26 April 2017 (UTC)

Hello Marti, thank you for your feedback. I will add more details for "methods and activities" and tell you when it is done.

For the tasks, I have spent more time on some core features and cut some secondary improvements, I hope this is fine, more details in Grants:IEG/Wikiscan_multi-wiki/Final#Actual_spending. --Akeron (talk) 08:43, 8 May 2017 (UTC)

@Mjohnson (WMF): I have updated Grants:IEG/Wikiscan multi-wiki/Final. Akeron (talk) 14:33, 9 June 2017 (UTC)