User:MPopov (WMF)/Notes/DevOps

An entry in a series of notes on setting up environments (e.g. User:MPopov (WMF)/Notes/Setup)

Puppet

Files, roles, profiles, and other modules

data.yaml

used by ops engineers to manage users, system users, groups, and access/permissions

profile::analytics::cluster::packages::statistics

specific R, Python, and system packages that should be installed on stat hosts (no Hadoop client related packages)

profile::analytics::cluster::packages::common

for packages/libraries which need to be installed on all nodes in the analytics cluster

role::statistics::explorer

statistics & analytics cluster explorer (all stat100X hosts)

profile::statistics::explorer::misc_jobs

for managing miscellaneous jobs to be run on a statistics::explorer node

statistics::discovery

manages cron jobs and /srv/discovery directory related to legacy Search Platform (formerly Discovery) metrics, including external traffic and WDQS metrics as part of Discovery Dashboards (partially decommissioned)

profile::discovery_dashboards

set of profiles used in corresponding roles for provisioning CloudVPS instances

r_lang

this module enables Puppet-managed installation of R packages from CRAN, Git, GitHub, and Bioconductor

shiny_server

this module creates a service shiny-server that serves Shiny applications from /srv/shiny-server through port 3838

Example: simple server with Wikimedia CloudVPS

After launching a new instance with Horizon (see FAQ for more information), apply role::simplelap to it in the Puppet Configuration to have an Apache-based server with PHP. BTW: role::simplelamp2 is for a server that has MariaDB.

Once the role has been applied to the instance, provisioning can be initiated manually with sudo puppet agent -t

An applied example is the MEP index.

Resources and references

• Pulling ‘Puppet’ strings on Discovery’s Dashboard framework
• Writing puppet code – how to write it, when to write it, where to put it

MediaWiki Vagrant

Working on MediaWiki Core or extensions like EventLogging and WikimediaEvents requires testing with MediaWiki Vagrant ("mw-vagrant"). Following these instructions for setting up, including downloading and installing Vagrant and VirtualBox. The Roles page has an outdated listing of the available roles.

After enabling eventlogging and wikimediaevents roles, add npm::node_version: 10 to puppet/hieradata/common.yaml to use NodeJS 10 (required for EventLogging, EventGate, EventBus).

To update MediaWiki (since vagrant git-update may not always work:

$> vagrant ssh
vagrant> cd /vagrant/mediawiki
vagrant> git pull
vagrant> composer update
vagrant> foreachwiki update.php --quick
vagrant> exit
$> vagrant reload --provision

Example: EventLogging development

First, enable the role and provision the VM:

vagrant roles enable eventlogging --provision
cd mediawiki/extensions/EventLogging
npm install

After that git checkout -b <feature-name>, etc.

Example: analytics instrumentation development

First, enable the role and provision the VM:

vagrant roles enable wikimediaevents --provision
cd mediawiki/extensions/WikimediaEvents
npm install

Also, update the secondary event schemas repository:

cd srv/schemas/event/secondary
git pull
npm install

Follow these instructions for initializing & materializing the example schema "analytics/link_hover". For local development, since mw-vagrant doesn't use mediawiki-config (see below), streams can be configured by creating $wgEventStreams in mw-vagrant's LocalSettings.php:

$wgEventStreams = [
    [
        'stream'       => 'analytics.link_hover',
        'schema_title' => 'analytics/link_hover',
    ],
];

Test by going to: http://dev.wiki.local.wmftest.net:8080/w/api.php?action=streamconfigs&format=json

Go to http://dev.wiki.local.wmftest.net:8080/wiki/VagrantRoleRestbase and run the mouseover instrumentation example in browser console. Hovering over "mw:RESTBase" link should trigger the event.

Testing locally before submitting for review

Before submitting with git review, use automated testing by calling vagrant provision and navigating to http://dev.wiki.local.wmftest.net:8080/wiki/Special:JavaScriptTest

Also, from inside the directory (e.g. mediawiki/extensions/EventLogging, as in the example above): ./node_modules/grunt/bin/grunt eslint to do linting, which is separate from the automated tests (in tests/)

For manual testing, create a new account to use because the following command (to be run in browser console) creates a hidden preference for displaying on-page notifications when events are logged:

mw.loader.using('mediawiki.api.options')
    .then(
        () => new mw.Api().saveOption('eventlogging-display-web', '1')
    );

Alternatively, EventLogging's debugMode can be enabled for a page view with mw.user.options.set( 'eventlogging-display-web', 1);

For more information, see EventLogging/Programming

MediaWiki Config

Wiki-specific production configuration lives in mediawiki-config/wmf-config/InitialiseSettings.php. At times it may be useful to access settings, or to export them to a portable format (e.g. JSON) that can be read into R or Python. This can be accomplished with the following PHP code:

require_once('mediwiki-config/src/defines.php');
require_once('mediawiki-config/wmf-config/InitialiseSettings.php';

$wmf_settings = wmfGetVariantSettings();

$json_encoding = json_encode($wmf_settings, JSON_PRETTY_PRINT);

$fp = fopen('settings.json', 'w');
fwrite($fp, $json_encoding);
fclose($fp);

To extract and export a particular setting, replace $wmf_settings with $wmf_settings['wgEventStreams'] (for example).