Wikimedia Blog/Drafts/Quantitative versus Qualitative: More friends than enemies

This was a draft for a blog post that has since been published at http://blog.wikimedia.org/2014/09/24/quantitative-versus-qualitative-more-friends-than-enemies/

Title ideas

• Quantitative versus Qualitative: More friends than enemies
• ...

Body

As Wikimedia program leaders and evaluators work together toward more systematic measurement and evaluation strategies, we face a common challenge to obtain both systematic quantitative data and qualitative information. By doing so, we can establish successful practices in outreach programs and understand the depth and variety of Wikimedia programs across our vast Wikimedia landscape. To meet this challenge, we are working to combine efforts and knowledge for two key purposes:

• To generalize program knowledge and design patterns of Wikimedia programs and
• To deepen our understanding of Wikimedia programs and projects and how they impact our communities.

Program leaders should seek a balance between quantitative and qualitative information.

Oftentimes, program leaders and evaluators question whether methods and measures that are quantitative are preferred over the qualitative, or whether qualitative outcomes can be given value like quantitative outcomes.

A good evaluation requires numbers and stories - one does not have meaning without the other. How those stories and numbers are collected can - and will - vary greatly, as each program leader seeks to understand their project or program story. Methods and measures for program evaluation should be designed to track how your project or program is doing, what it intends to do and how well the expected outcomes are reached. Whether those methods and measures are qualitative or quantitative will vary depending on your interests, your measurement point and your evaluation resources, but should, no matter what, be useful in telling the story of what you did and whether or not your work was a success. (Read more on the Evaluation portal.)

Most often, through triangulation of quantitative and qualitative measures, today’s social researchers define approximate measures or «proxies» for focusing on and measuring a phenomenon of interest. Through triangulated measures, qualitative and quantitative information can tell a better story of outcomes than either can alone. For instance, consider the phenomenon of volunteer editing behavior:

QUANTITATIVE "edit count" "bytes added/removed" "page views"		A BETTER STORY
+	=
QUALITATIVE "article subjects" "categories" "quality" ratings

In a mixed-methods world, quantitative and qualitative tend to be two sides of the same measurement coin: related and nearly inseparable in practice.

How do numbers and qualitative attributes come together?

All quantitative measures are based on qualitative judgments; all qualitative measures can be coded and analyzed quantitatively.

As illustrated in the images below, numbers do not mean anything without assigning a description; anecdotes mean nothing without numbers.

Whether it is a question about physical count data, or about an attitude, we must create the meaning of numbers, and numbers with meaning in measurement. (Read more on a detailed example of how to ask yourself these questions during an education program with students, leading to a combined quantitative and qualitative approach.)

Defining count data in ordinal response categories that are assigned a response numeral (1) through (7)

Assigning numbers to mean different levels of applicability of a sensed state of being which cannot be otherwise observed.

Converting qualitative to quantitative via count data using the sum of codeable observations.

Next steps for Program Leaders

Trying to choose the best measures for your Wikimedia project or program?

Check out the helpful Measures for Evaluation matrix of common outcomes by program goal. We are working to map measures and tools for it.

Last week, the Program Evaluation and Design team initiated the Foundation’s second round of voluntary programs reporting. We invite all program leaders and evaluators to participate yet again in the most epic data collection and analysis of Wikimedia programs we've done so far. This year we will examine more than ten different programs:

Editing workshops On-wiki writing contests Editathons Wikipedia Education Program Conferences

Wikimedian in Residence GLAM content donations Wiki Loves Monuments Hackathons Wiki Loves Earth, Wiki Takes, Wiki Expeditions, and other photo upload events

Did you lead or evaluate any of these programs during the time from September 2013 through September 2014? If so, we need your data! For the full announcement visit our portal news pages.

Reporting is voluntary, but the more people do it, the better we can represent programs. This voluntary reporting will help us understand the depth and impact of programs across different contexts. It allows us to come together and generate a bird’s eye view of programs so that we can examine further what works best to meet our shared goals for Wikimedia. Together we can grow the AWESOME in Wikimedia programs!

Jaime Anstee, Ph.D., Program Evaluation Specialist, Wikimedia Foundation

Notes

Category: Program evaluation and design Tags: Wikimedia learning, research, qualitative, quantitative Ideas for social media messages promoting the published post:

Twitter (@wikimedia/@wikipedia):

(Tweet text goes here - max 117 characters)
How do you bring numbers and qualitative attributes together? Find out more of evaluating in a mix-method world:/

Facebook/Google+

• Wikimedia Program Evaluation goes deep in the differences between qualitative and quantitative information. How to bring them together in evaluation and reporting? Find out more of how to evaluate and report using mix methods.