Learning patterns/Hiring a metrics processor for a Wikimedia outreach project

A learning pattern fororganizational

Learning Pattern Library

Hiring a metrics processor for a Wikimedia outreach project

problemThe Wikimedia Foundation requests "global metrics" reporting when providing grant funding for Wikimedia outreach. Learning to report global metrics is its own skill set, as is making this data locally relevant.

solutionFor some outreach projects, it is worthwhile to consider hiring a metrics processor.

creator• Bluerasberry

created on16:11, 30 March 2016 (UTC)

status:DRAFT

What problem does this solve?

"Wikimedia global metrics" refers to the data which must be reported by all grant recipients of Wikimedia funding if they do outreach projects. If a project is sure that its participants are unable to process metrics themselves, here are options for seeking metrics processing services.

What is the solution?

Consider the scope of the work Consider hiring strategies Consider the pay rate

Things to consider

Hiring process

Hiring process Consider the following hiring strategies:

Direct hire of a local worker

Advantages:

1. Greatest likelihood of interaction between hired worker and local Wikimedia chapter participants
2. Most freedom to request any sort of labor
3. Follows history of Wikimedia chapters doing direct hire
4. Greatest opportunity for innovation for integrating metrics processing with other chapter projects
5. Continues whatever local precedent is established for doing any event reporting
6. Continues whatever local precedent is established for sorting and organizing reports

Disadvantages:

1. Highest chance of drama, meaning community criticism of work output and need for worker response
2. Freedom in hiring means greatest difficulty in keeping the role narrow and achievable
3. Increases the pressure to hire someone who is active in chapter and has data processing skill set, and very likely there would be few applicants meeting that description
4. Budget is for 10 hours/week. This is a difficult sort of hire for such specialized work.

Subcontract through a local nonprofit incubator, university, or similarly experienced support organization

Advantages:

1. Allows the chapter to treat the hire as a financial transaction with the least emotional context
2. Easy to fire or replace worker if position does not work
3. Highest potential for well-defined relationship
4. Strongest potential for requesting work on a schedule
5. Higher potential to be able to hire for a particular skill set
6. Higher potential to be able to hire someone with experience doing nonprofit grant reporting
7. Gain the benefit of professional mentoring over hired worker, who would have their own professional support at the host institution

Disadvantages:

1. Less social exchange between worker and Wikimedia chapter
2. Less potential for ongoing relationship between chapter and incubator - this is an “incubation” relationship
3. Less flexibility to change labor demands mid-project; the workflow would be set in the beginning
4. Almost no chance of hiring anyone with Wikipedia familiarity, so all data and reporting will be reduced to standard nonprofit management
5. Little chance of innovation in the role - work will be done as requested and no more
6. Most likely to raise the issue of subcontracting, which is difficult to manage and report on taxes

Purchase consulting services from another Wikimedia chapter

Advantages:

1. Gain backing of experienced organization which can meet WMF grant reporting requirements.
2. This is the only certain way to hire an experienced worker who already knows the job
3. Least chance of debate about pay rate - the other Wikimedia chapter already has this set at a price supported by local community
4. Wikimedia community trust is the best assurance that work meeting WMF standards will be done, or otherwise, partner chapter will share in blame and make things right somehow
5. Develops inter-chapter communication and partnerships, which would be very useful
6. Concentrates staff doing same work in one place
7. Gain management support over the employee, as the other Wikimedia chapter will have management staff to oversee them.
8. Likely to be possible to pay for consulting services without having do accounting or reporting for an employee or subcontractor

Disadvantages:

1. Remote worker will have limited interaction with local Wikimedia community
2. Funding leaves the local community

Purchase remote consulting services from an experienced Wikipedian in another country

Advantages:

1. Likely to gain the benefits of hiring someone at a Wikimedia chapter except for the staff management oversight
2. Less expensive than hiring someone at a Wikimedia chapter, because less administrative overhead

Disadvantages:

1. Hiring a freelancer is risky; if something goes wrong they do not have institutional backing
2. Same disadvantages as with hiring from a chapter

Possible workflows

As of early 2016 the work associated with this job has not been discussed in the Wikipedia community. The role could be executed in any of the following ways and might include scopes of workflow that include the following ideas:

1. Support for event organizers (both local and remotely) to process their own metrics. This would be the most conventional way to manage the position, because the conventional belief is that event organizers can manage metrics. The role of “metrics processor” was imagined to relieve burden from event organizers, and in creating this position, WM NYC seeks to go against conventional practice and centralize the work. Still - the traditional way to imagine the role is in the context that “anyone ought to be able to do this”, because that is the idea that until early 2016 was the most shared.
2. Capture event dates and participant lists, then run Wikimetrics. This is the minimum that the Wikimedia Foundation wants. One problem with doing only this is that there would be no recognition of precision or accuracy without subjective notes being taken following interviews with the event coordinators about how many people have participated in events, when event support ended, and what participation was supposed to mean. Also Wikimetrics alone does not categorize events by theme or sponsor.
3. Do #2, but then also, download all of the metrics from Wikimetrics and somehow adapt them into a single qualitative comprehensible narrative for the Wikimedia community.
4. Do #3, but then also, adapt the single big picture narrative into a series of shorter more focused stories for interest groups. Interest groups might mean “all AfroCROWD events”, or “all Art+Feminism events”, or “all events at the Museum of Modern Art” or any other venue.
5. Do #2, then download all of the metrics from Wikimetrics and somehow do statistical analysis on the numbers. For a group doing 50 events in a year, then most probably there would be more than 100-300,000 data points in a year. This information could be utilized in the Strategic Plan for the organization managing the metrics processing as well as be provided to Wikimedia projects globally.

Desired skills

Here is bluerasberry's vision for the duties in order of priority.

1. Be able to communicate effectively with stakeholders. The point person for the position will be the hiring manager at Wikimedia NYC. Secondary to that one person, the worker must regularly communicate with the events coordinator for Art+Feminism, AfroCROWD, and Wikimedia NYC.
2. Process data to meet WMF global metrics reporting requirements. This is the only actual grant funding requirement.
3. Be able to publish a nonprofit annual report that looks good. There might be two variations of this - reports that look good to Wikipedians, and reports that look good to external organizations which request them.
4. Develop a plan to improve the accuracy and precision of global metrics reporting, and somehow qualify the current state of accuracy and precisions. For context, it is probably true that at least 25% of event participants are not reported at events, so their contributions do not appear in data. We need to establish a norm for recognizing the extent to which data is accurate.
5. After doing all of the above, increase data processing. There probably is more work here than anyone can do. From the beginning, it will not be possible to do everything, so the final goal is to increase efficiency and process more.
6. Wikipedia community participation. This would be nice to have. It is a traditional Wikimedia community request to make. It may not be essential for processing data into Wikimedia global metrics but all Wikimedia community members request this in all staff roles.

Pay rate

The New York Times reports that USD $80,000 (meaning $38/hour, assuming 52 weeks of work and 40 hours a week) is the start of middle class in Manhattan. What Is Middle Class in Manhattan? This job requires at minimum the skill set commonly associated with middle class professionalism.

$38/hour is a fair estimate for the lowest reasonable pay rate for a local worker. In the grant request, $60/hour is requested. Because the hiring method is not certain, $60 is the upper limit of what the chapter might try, but if a hiring method is chosen where a lower rate would be appropriate, then money could be returned to the foundation.

It is not certain that any suitable candidate will be identified for $60/hour because many people with the obvious skill set to do this would not want to work at this job for this rate. Again, the hope is to advertise the job and get 3-5 reasonable applicants at least. This is not a pay rate high enough to guarantee that this will happen.

Starting with the assumption that this job does require the skill set associated with middle-class pay, here are some extra reasons why this job should pay a little more than the bottom rate:

1. The consultant rate for this sort of job to contract through a university is established as $60-90/hour.
2. There have been at least three paid Wikipedian positions for hire in New York City that have gone unfilled. All offered pay at the bottom of middle class, meaning $30-40/hour. These seemed like fun jobs, whereas this data position is not what most people would call “fun”. This job should pay more than the fun jobs to attract candidates.
3. This is a novel position. Whomever takes it will be doing work without an established protocol. The role requires critical thinking, communication, and experience beyond typical consulting roles.
4. This is a 10 hour/week position. Part time hires get higher pay than full time hires.
5. This is likely to be a stressful position where the person hired will take significant criticism. The person coming into the role has to be aware that their work will face community scrutiny in a way that is unusual as compared to data processing in other fields.
6. The rate for reporting at least one harassment incident within one year for Wikimedia staff and community leaders in the US must be greater than 50%. The person hired into this position needs to know that.
7. The commercial rate for consulting of this sort in New York is probably around $120/hour. 50% of the commercial market rate is reasonable if not low. This project has similar reach and liability to what WMF communications consultants do, and hires in that space are a bit outside the community and might be hired at commercial rates.

Endorsements

• The metrics & reporting position is a necessity for producing reports that satisfy WMF grant reporting requirements, for producing reports that will impress other organizations to take chapters and user groups seriously and commit to partnerships, for potential future fundraising, and for showcasing and evangelising the benefits of professional metrics to the entire Wikimedia community. Totally agree with the content, although I'm subjectively not sure the costs are high enough, at least in New York City. We really need to try this out to produce data on how well this approach works and what the costs will be. Becksguy (talk) 15:20, 31 March 2016 (UTC)