Learning and Evaluation/Evaluation reports/2013/GLAM content partnerships
Part of the Wikimedia Resource Center
Learning and Evaluation
This is the Program Evaluation page for GLAM content partnerships. It currently contains information based on data collected in late 2013 and will be updated on a regular basis. For additional information about this first round of evaluation, please see the overview page.
This page reports data from seven program leaders about nine GLAM content release partnerships. Program leaders who responded to the pilot survey about GLAM content release partnerships included program leaders affiliated with user groups or chapters, as well as those without any group affiliation.
Key lessons include:
- There was a high level of variation in the inputs and scope of the reported GLAM content release partnerships. For example, some content release partnerships cost nothing to implement, while one content partnership reported a $8,000 USD cost. Another example: one content partnership reported a project that consisted of uploading one file, when others uploaded thousands of files.
- Program leaders who engage in activities that lead to GLAM content release partnerships have five priority goals focused around contributions, quality, and increasing awareness and respect about the Wikimedia movement.
- On average, GLAM content release partnerships have four people involved and take an average of three months to implement from beginning to end.
- Only a small portion of the donated media have been used in Wikimedia project pages compared to the large amount of media that is donated in total.
- Most program leaders believe that their GLAM partnerships will continue, and just under a half are secure in believing that new partnerships will develop as a result of their content release partnerships.
- Program leaders who implement programmatic activities that lead to GLAM content releases are experienced, and produce documentation and blogs about their implementations, creating resources for others to adapt and implement for their own content release partnerships.
- Without surveying, we're unable to determine if qualitative priority goals are being met. We define qualitative priority goals as those such as increased awareness and respect for Wikimedia projects and increased support for the open knowledge/free content movement. We aim to support program leaders in examining these priority goals.
- Our current ability to do a meaningful return on investment analysis is limited. It raises questions around how to make the encyclopedic value of an image part of the equation.
Planning a GLAM Content Partnership? Check out some process, tracking, and reporting tools in our portal and find some helpful tips and links on this resource page
Program basics and history
[edit]
A GLAM content partnership is when a gallery, library, archive, or museum (also called GLAM) works with Wikimedia community members to upload media to Wikimedia Commons. The media, which is generally educational, artistic, or historic in nature, must be freely licensed or fall under the public domain. Wikimedians will partner with the interested GLAM institution, and guide them through the steps for content partnership: (1) make sure the collection of media to be "donated" is properly licensed, (2) license that media if needed, (3) prepare the metadata, (4) upload the media to Wikimedia Commons, and (5) distribute the media throughout Wikimedia projects, such as Wikipedia.
Many content release partnerships are led by chapters or affiliates in partnership with specific GLAM staff members. Wikipedians in Residence may also serve as program leaders for content release partnerships.
One of the first content release partnerships by a cultural institution took place in 2008, when Wikimedia Italia signed an agreement with the Palladio Museum to celebrate the 500th anniversary of the birth of architect Andrea Palladio. Wikimedia Italia worked with the museum to create and improve content on Wikipedia related to Palladio's work, and the museum contributed 87 images to Wikimedia Commons.[1]
After a cooperation between the German Federal Archive ("Bundesarchiv") and Wikimedia Deutschland[2] and a cooperation between the Tropenmuseum and Wikimedia Nederland generated a lot of media and community attention in late 2008 and early 2009, programmatic activities around GLAM content release partnerships accelerated. And even though the partnership between the German chapter and the Bundesarchiv, which had once been described by some as a "textbook example of a cooperation between Wikipedia and a public authority,"[3] ended informally in 2010 (with the Bundesarchiv being disappointed about the many cases of non-license-conform re-usage of its images by third parties[4]), cultural institutions are forming new cooperations with Wikimedia entities until this day.[5]
Report on the submitted data
[edit]Response rates and data quality/limitations
[edit]Seven program leaders reported data about nine GLAM content release partnerships that took place between February 2012 and October 2013. GLAM content partnership reporting was harder to collect because of the extreme variability in the budget and hour inputs and the additional variances in the number of files uploaded per donation. As with all the program report data reviewed in this report, report data were often partial and incomplete. Please refer to the notes, if any, in the bottom left corner of each graph below.
Report on the submitted data
[edit]Priority goals
[edit]- According to respondents, GLAM content release partnerships have five priority goals.
We asked program leaders to select their priority goals for programmatic activities that lead to GLAM content release. We provided 18 priority goals with an additional 19th option to report on "other" goals. They could select as many or as few as they wanted.
All seven of the program leaders who reported on GLAM content release partnerships selected priority goals for their nine GLAM content release partnerships. Those seven selected between three and thirteen priority goals for each of their GLAM content release partnerships.[6] Our team noted five stand-out goals that appeared as priorities among the reporting program leaders (see table below):
Inputs
[edit]In order to learn more about the inputs that went into planning GLAM content release partnerships, we asked program leaders to report on:
- The budget that went into planning and implementing the GLAM content partnership.
- Any donations that they might have received for the event: a venue, equipment, food, drink, giveaways, etc.
- The hours' that went into planning and implementing the the GLAM content partnership.
Budget
[edit]- Just over half of the GLAM content release partnerships reports included a budget.
Out of reported nine GLAM content release partnerships, five (56%) reported a budget, but three of those five budgets reported were zero. In total, budgets ranged from $0 US to $8,000 US. [7]
Donated resources
[edit]- A minority of program leaders reported donated resources for their GLAM content partnership, which is no surprise, since donated resources aren't required for this type of program.
GLAM content release partnerships don't necessarily require additional resources outside of a potential budget and time put into implementing the program, but, some program leaders did report receiving donated resources. Meeting space (33%) was the most commonly reported donated resource, followed by equipment (22%) (see Graph 1).
Hours
[edit]- The majority of program leaders were able to report how many staff and volunteer hours went into implementing their GLAM content release, with the majority of programmatic activities being implemented by volunteers. Volunteers put an average of 60 hours of time into implementing a GLAM content release.
Staff and volunteers put the following time into implementing programmatic activities that led to GLAM content release partnerships, according to respondents:
- 78% of program leaders reported staff hours, which ranged from 0 to 50 hours with an average of 0 hours.[8]
- 78% of program leaders reported volunteer hours, which ranged from 20 to 200 hours with an average of 60 hours.[9]
- Total hours reported ranged from 20 to 200 with an average of 80 hours per donation [10]
With the average volunteer time being 60 hours, and staff time being zero hours, we can draw the conclusion that volunteers are taking the lead at implementing GLAM content releases.
Outputs
[edit]We also asked about two outputs in this section:
- How many people participated in the programmatic activity that led to the GLAM content partnership?
- How many hours did it take to achieve the result of a GLAM institution to donate its content?
Participation
[edit]- The average GLAM content release partnership has four people involved in the implementation process, however, it's unclear who those people are without further data collection.
Out of all the GLAM content release partnerships reported in this evaluation, the number of participants ranged from one to nine with an average of four.[11]
Additional research needs to be done to learn who these participants are: Wikimedia community volunteers and staff, and/or volunteers and staff from GLAMs.
Implementation time
[edit]- According to reported data, a GLAM content release takes an average span of three months from initiation to completion. Additional program leader reporting is required to better understand how the activities and hours input into a content release partnership are spread over this time.
There are many events and activities that may be involved in implementing a content partnership from convincing a GLAM to freely license material, standardize metadata, upload the images with volunteers, and distribute those images on Wikimedia projects. Five program leaders (56%) reported the length of time it took to implement their content partnership from inception to completion. Implementation windows spanned from 8.5 weeks to 35 weeks, with an average of three months[12]. Remembering that the actual hours invested over these months averaged only 80 hours per donation (see "Hours" above), further investigation is required to learn more about the types of events and activities take place over the course of implementation.
Outcomes
[edit]Content production and quality improvement
[edit]- Only two GLAM content releases[13] reported budgets that were not zero. For these two, the cost per item uploaded was $1.56 USD and $2.96 USD.
Two GLAM content releases reported budgets that were not zero. This allowed us to examine how much each uploaded file (image, video, etc.) cost to donate to Wikimedia Commons. The cost per item uploaded was $1.56 USD for one donation and $2.96 USD for the other donation with a reported budget. We need more data from additional implementations with non-zero budgets, which include value estimates for the hours input into the work (i.e. staff who digitize images at GLAMs), in order to learn more about the actual investment per file for GLAM content releases.
- The number of participants and implementation hours does not relate directly to the number of files uploaded as a result of a GLAM content partnership.
Seven out of the nine reports (78%) included how many hours it took to implement the program as well as how many participants were involved in the implementation, and how many files were donated as a result. The number of files donated varied greatly from 1 to 18,421 files. The amount of time and number of participants does not appear to relate to how many files get donated and uploaded (see Graph 2).[14]
- On average, it takes a little over six minutes of input time per photo or media file uploaded as part of a content partnership.
We were also able to estimate how much time was invested into the donation process based on input hours program leaders provided us. Dividing the hours invested by the number of photos added, the cost per photo or media added ranged from 14.4 seconds (.004 hours) to 36.6 minutes (.61 hours), with an average of 6.6 minutes (.11 hours) per photo/media page added[15] (Graph 3).
-
Graph 2. Hours to Photos/Media Added. As illustrated in the graph, number of photos/media added ranged from 1 to 18,421 after inputs ranging from 20 to 200 hours. In the graph, the number of volunteers and GLAM staff working on the project are illustrated by bubble size and label. For two donations for which there was no report of input hours, the median of 80 hours was used in order to include them in the graph. It appears that, neither the number of participants, nor the amount of hours invested, were correlated with the number of photos/media added in the donation. -
Graph 3. Hours to images/media added. As illustrated in the box plot, content release partnerships demonstrated a wide range of variation in the input hours to the amount of content added; from as little as a quarter of a minute per image/media added to 36.6 minutes invested per image/media added. The median reported input time was 6.6 minutes per image/media added with a distribution in which half of the reports surrounding the median ranged from 0.7 minutes to 24.0 minutes.
We wanted to investigate the quality of the donated images/media to learn two things:
- Were the donated images/media being used to improve Wikimedia project pages, such as Wikipedia articles?
- Were the donated images/media being nominated and approved as Featured pictures or Valued images?[16]
- Only a small percentage of the total reported GLAM images/media files donated were used in Wikimedia projects. An even smaller amount of those files are considered Valued images by the community, and zero are Featured images.
Across the nine reported GLAM content releases, six contributed 27,434 pages and 26,791 images/media files to Wikimedia Commons. Pages may include wiki pages focused around organizing the donation and processing the files, file upload pages, or pages on Wikipedia. Out of the total uploaded images/media, 1,721 unique files (6%) were used on Wikimedia project pages (see Graph 4). Only 112 (less than 1%) of those files were nominated and approved as valued images. No featured images were reported.
Recruitment and retention of new editors
[edit]In order to learn about recruitment and retention, we asked two things:
- Will the GLAM involved in the reported GLAM content partnership continue to partner with Wikimedia?
- Could this implementation result in new partnerships with other GLAMs?
- The majority of program leaders expect their GLAM partnerships to continue, and little under half believe that new partnerships will likely develop.
Program leaders shared with us that (see Graph 5):
- 67% anticipated that the GLAM partners developed through their content partnership would continue a relationship with Wikimedia.
- 44% indicated the project would likely lead to other GLAM partnerships.
-
Graph 5. Recruitment and retention. For all content release partnerships reported, the majority of program leaders indicated that GLAM partners were likely to continue a relationship with Wikimedia and also might lead to other GLAM partnerships, with little under half believing that partnerships will absolutely continue. This reflects at least the perception of progress toward priority goals of increasing awareness of Wikimedia projects and growing support for open knowledge/free license.
Replication and shared learning
[edit]All of the programs leaders who reported GLAM content release partnerships consider themselves experienced enough to assist others in implementing their own program of this type. The majority also reported having produced blogs or other online information to share with the public.
Out of the nine program implementations reported by program leaders (see Graph 6):
- 100% of the GLAM content release partnerships reported were implemented by an experienced program leader who could help others conduct their own GLAM content release.
- 78% had blogs or other online information available for others to learn from.
No program leader reported developing how-to guides for their GLAM content partnership; however, such materials have previously been created by other members of the community and do exist.
-
Graph 6. Replication and shared learning. The biggest strengths that GLAM content release partnerships demonstrated in terms of potential replication and shared learning were that all the donations were led by an experienced program leader who could help to guide others and the majority had blogs or other online documentation to share learnings with others.
Summary, suggestions, and open questions
[edit]
How does the program deliver against its own goals?
[edit]The value of content released from cultural institutions for Wikipedia is obvious. Images are one of the backbones of an encyclopedia. Without the hundreds of thousands photos that have been released by cultural institutions over the last couple of years, large numbers of our articles would lack pictorial illustration. Many of the images added through GLAM partnerships would be unavailable by other means, making them particularly valuable. Over the last couple of years, GLAM content release partnerships have been a continuous stream of newly uploaded images on Wikimedia Commons:
| Year | Total amount of images uploaded to Wikimedia Commons through GLAM content released |
|---|---|
| 2008 | 82,713 |
| 2009 | 145,229 |
| 2010 | 37,485 |
| 2011 | 141,974 |
| 2012 | 25,076 |
| 2013 | 469,899 |
With regard to the other goals that GLAM content partnership program leaders reported back, there is a need for further investigation. At this point it is unclear which program mechanics different chapters have put in place in order to reach goals like "increasing support for the free content movement" or "increasing awareness of Wikimedia projects". And it is also unclear how and to which extent chapters have been measuring their success in these areas.
How does the cost of the program compare to its outcomes?
[edit]With regard to any kind of return on investment analysis, our first look at GLAM content release partnerships through the program evaluation lens quickly reaches its limits. It raises questions about how to make the encyclopedic value of an image part of the equation. Is a large image donation more valuable for Wikipedia than a smaller one, just because larger amounts of images have been uploaded under a free license? What about those image donations that are smaller in size but contain a larger percentage of highly valuable images (i.e,. images that are now part of articles that are being accessed many thousands of times per month)? It seems like in these cases, content release partnerships can only be appraised correctly if we take the click-rates for the corresponding Wikipedia articles into account. Also, what about rare photographs or valued images, e.g. images of subjects that can't be replicated anymore (for various reasons, e.g. photos of extinct species or one-of-its-kind images of demolished historic monuments)? How can the value of such a photograph be ever appropriately weighted as part of a return on investment analysis?
Whereas the execution of most other programmatic activities doesn't require the implementers to have any formal affiliation with a chapter, cultural partnerships with GLAM institutions strongly benefit from the official backing by a formal Wikimedia institution. Anecdotal evidence indicates that having an affiliation with a Wikimedia Chapter and being able to prove this affiliation (e.g., by showing a business card) helps Wikimedia volunteers and staff members build trust with potential partners.
How easily can the program be replicated?
[edit]Looking at the existing documentation of GLAM content partnerships, a large number of case studies describing individual partnerships exist, which is not surprising given the good overall status of GLAM materials assembled by volunteers over the last couple of years.[17] Highly commendable are documents like the existing Guide to content partnerships, which keeps new program implementers from re-inventing the wheel.
Next Steps
[edit]- Next steps in brief
- Increased tracking of detailed budgets and donated resource inputs by program leaders.
- Improved ability for program leaders to collect and track GLAM staff, chapter/affiliate staff, and volunteer input hours.
- Increased tracking of media uploaded, media quality, and media usage by program leaders.
- Improve the design of GLAM content partnerships to better meet priority goals related to awareness of Wikimedia projects and increased support for the open knowledge/free content movement.
- Surveying of GLAMs to see if partnership has met qualitative priority goals related to awareness of Wikimedia projects and increased support for the open knowledge/free content movement.
- Next steps in detail
As with all of the programs reviewed in this report, it is key that efforts are made toward properly tracking and valuing programming inputs in terms of budgets, donated resources, and hours invested as well tracking participants of content partnerships, user names, content partnership activities, and event dates for proper monitoring and a better understanding of the overall process. Further investigation of expectations and efforts directed toward other goal priorities including the top two goals of: increasing awareness of Wikimedia projects and increasing support for the open knowledge/free content movement. It will be important to more clearly articulate programming activities specifically aligned to those goals and to develop strategies for measuring such outcomes. Next steps should also include better tracking of the quality of donated material (valued images/featured pictures) and its use in Wikimedia projects.
Appendix
[edit]Summative Data Table: GLAM Content Donations (Raw Data)
[edit]| Percent Reporting | Low | High | Mean | Median | Mode | SD | |
|---|---|---|---|---|---|---|---|
| Non-Zero Budgets | 33% | $3,800.00 | $8,000.00 | $5,900.00 | $5,900.00 | none | $2,969.85 |
| Staff Hours[18] | 78% | 0.0 | 50.0 | 8.6 | 0.0 | 0.0 | 18.7 |
| Volunteer Hours | 78% | 20.0 | 200.0 | 72.9 | 60.0 | none | 59.9 |
| Total Hours | 78% | 20.0 | 200.0 | 81.4 | 80.00 | 80.00 | 56.7 |
| Donated Meeting Space | 33% | Not Applicable - Frequency of selection only | |||||
| Donated Materials/ Equipment | 22% | Not Applicable - Frequency of selection only | |||||
| Donated Food | 0% | Not Applicable - Frequency of selection only | |||||
| Donated Prizes/Give-aways | 0% | Not Applicable - Frequency of selection only | |||||
| Participants | 100% | 1 | 9 | 4 | 4 | 2 and 6 | 3 |
| Dollars to Participants | Not Applicable | ||||||
| Input Hours to Participants | Not Applicable | ||||||
| Bytes Added[19] | 22% | 0 | 178225 | 89113 | 89113 | none | 126024 |
| Dollars to Text Pages (by Byte count) | 0% | No Data | |||||
| Input Hours to Text Pages (by Byte count) | 0% | No data | |||||
| Photos/Media Added[20] | 100% | 1 | 18421 | 3827 | 1284 | none | 6685 |
| Dollars to Photos | 56% | Only 2 reporting non-zero budget: Low=$1.56/photo and High=$2.96/photo | |||||
| Input Hours to Photos/Media[21] | 67% | 0.004 | .61 | 0.30 | 0.11 | none | .41 |
| Pages Created or Improved | 44% | 0 | 350 | 107 | 39 | 0 | 166 |
| UNIQUE Photos Used[22] | 100% | 1 | 1393 | 287 | 93 | none | 544 |
| Dollars to Photos USED (Non-duplicated count) | Only 1 with non-zero budget: $64.52/Unique Photo Used | ||||||
| Input Hours to Photos USED (non-duplicated count) | 56% | $0.00 | $1.18 | $0.47 | $0.45 | none | $0.48 |
| Good Article Count | 44% | 1 | 33 | 7 | 0 | 0 | 15 |
| Featured Article Count | Not applicable | ||||||
| Quality Image Count | Not applicable - None reported | ||||||
| Valued Image Count | 44% | 1 | 404 | 179 | 156 | none | 171 |
| Featured Picture Count | Not applicable - None reported | ||||||
| 3 Month Retention | Not applicable - 67% reported GLAM would continue Wikimedia relationship AND 44% reported this partnership would lead to other GLAM partnerships | ||||||
| 6 Month Retention | |||||||
| Percent Experienced Program Leader | 100% | Not Applicable - Frequency of selection only | |||||
| Percent Developed Brochures and Printed Materials | 0% | Not Applicable - Frequency of selection only | |||||
| Percent Blogs or Online Sharing | 78% | Not Applicable - Frequency of selection only | |||||
| Percent with Program Guide or Instructions | 0% | Not Applicable - Frequency of selection only | |||||
Bubble Graph Data
[edit]| Report ID | Photos/Media Added[23] | Total Hours | Number of Participants |
|---|---|---|---|
| 1 | 417 | 80 | 5 |
| 2 | 200 | 200 | 4 |
| 3 | 33 | 20 | 2 |
| 8 | 1819 | 50 | 6 |
| 12 | 18,421 | 80 | 6 |
| 24 | 95 | 80 | 1 |
| 36 | 1 | 80 | 9 |
| 37 | 1284 | 3460 | 3 |
| 69 | 5138 | 80 | 2 |
More Data
[edit]| Report ID | Program Length (weeks) | Budget | Staff Hours | Volunteer Hours | Total Hours | Donated Space | Donated Equipment |
|---|---|---|---|---|---|---|---|
| 1 | 0 | 80.00 | 80 | Yes | |||
| 2 | 0 | 200.00 | 200 | Yes | |||
| 3 | $0.00 | 0 | 20.00 | 20 | |||
| 8 | 12.00 | $0.00 | 0 | 50.00 | 50 | ||
| 12 | 12.00 | 50 | 30.00 | 80 | |||
| 24 | 8.50 | ||||||
| 36 | $0.00 | 0 | |||||
| 37 | 21.00 | $3,800.00 | 3400 | 60.00 | 3460 | Yes | Yes |
| 69 | 35.00 | $8,000.00 | 10 | 70.00 | 80 | Yes |
| Report ID | Number of Participants | Photos/Media added | Number Other Pages Created |
|---|---|---|---|
| 1 | 5 | 0 | 417 |
| 2 | 4 | 0 | 200 |
| 3 | 2 | 33 | |
| 8 | 6 | 1819 | |
| 12 | 6 | 18,421 | |
| 24 | 1 | 95 | |
| 36 | 9 | 1 | 10 |
| 37 | 3 | 1284 | 16 |
| 69 | 2 | 5138 | 0 |
| Report ID | Number UNIQUE Photos Used | Number Valued Images | Number Featured Pictures | Number Pages Created or Improved | Recruitment and Retention - Will GLAM continue Wikimedia relationship | Recruitment and Retention - Will this project lead to other GLAM partnerships (i.e. with other GLAMs, with other content) |
|---|---|---|---|---|---|---|
| 1 | n/a | Yes | Maybe | |||
| 2 | n/a | No | Maybe | |||
| 3 | 17 | Yes | Maybe | |||
| 8 | 111 | 111 | Maybe | Maybe | ||
| 12 | 1393 | 3 | Yes | Yes | ||
| 24 | 75 | 75 | Yes | Yes | ||
| 36 | 1 | 1 | 0 | 350 | Yes | Maybe |
| 37 | Maybe | Yes | ||||
| 69 | 124 | 0 | Yes | Yes |
Notes
[edit]- ↑ This is believed to have been the earliest formalized GLAM content partnership, according to crowdsourced reporting from community members. A case study was recently produced about this partnership, which can be read on Wikimedia Italia's website.
- ↑ As documented in a case study Fallstudie Bundesarchiv, also available in English as German Federal Archives case study.
- ↑ By Tagesschau, the German national and international television news service.
- ↑ See Ins Netz gegangen - Dr. Oliver Sander: Die Kooperation des Bundesarchivs mit Wikimedia, presentation by Oliver Sander of the German Federal Archives, on YouTube.
- ↑ For a more comprehensive list see Commons:Partnerships.
- ↑ Mean= 7, Standard deviation= 3
- ↑ Mean= $5,900, SD= $2,969.85
- ↑ Mean = 8.6, SD= 18.7
- ↑ Mean= 72.9, SD= 69.9
- ↑ Mean= 81.4, SD= 56.7
- ↑ Mean= 4, SD= 3
- ↑ Mean = 15.9, SD = 10.8
- ↑ Note: Although "content production" is a direct product of the program event itself and technically a program output rather than outcome most of the program leaders who participated in the logic modeling session felt this direct product was the target outcome for their programming. To honor this community perspective, we include it as an outcome along with quality improvement and retention of "active " editors.
- ↑ Program leaders reporting content releases were also asked if the donation included non-media pages, such as information pages; 56% reported donations of this kind, including two donations that were only information pages, not media files.
- ↑ Mean= .30 hours, SD= .41 hours
- ↑ Note that GLAM content partnership images are not eligible for "Quality image" ratings.
- ↑ See the GLAM section on the Outreach wiki.
- ↑ Only three of reported non-zero staff hours.
- ↑ Only two reports of bytes added.
- ↑ Two GLAM content releases were in information pages only (no photos); those have been excluded from the summative data.
- ↑ Computed based on six donations that reported non-zero hour with photos added reports (includes two information page donations).
- ↑ Note: "Photos used" reference is to unique, non-duplicated, count of photos used as of Nov. 4, 2013.
- ↑ Note: For reports 1 and 2, the added pages reported are information pages, which is what the donation was, rather than photos or other media.