Barriers and solutions to assessing digital library reuse: preliminary findings

Introduction and problem statement

Current assessment efforts that focus on the unique content found in digital libraries are not meeting the needs of the practitioners that serve them, or the communities that use them[1]. Although numerous quantitative ways to view digital object use exist, from page views, to downloads, to newly developed multimedia standards for institutional repository use[2], these measurements do not reliably demonstrate the value of these materials in ways that resonate with the digital library community. Specifically, they are too often unfavorably, and perhaps unfairly, compared to very different types of library use, such as journal article downloads, chapter views or circulation statistics.

A key indicator of the impact of digital collections is content reuse, specifically how materials are utilized and repurposed. There have been ongoing efforts within the digital library community to demonstrate and measure reuse. These types of investigations have produced meaningful results, but efforts have been scattered among institutions and organization types. Additionally, in this relatively new arena, few recommended practices or gold standards have emerged. The lack of reuse metrics, combined with the lack of community assessment norms, has a two-fold effect: it makes it difficult for institutions to develop, using appropriate data, strong infrastructures and collections that are responsive to user needs; and it suppresses the ability of a digital library to demonstrate its value to stakeholders, including administrators, funding bodies and potential users.

An Institute of Museum and Library Services (IMLS) national leadership grant, “Developing a Framework for Measuring Reuse of Digital Objects” (Thompson et al., 2017), attempts to address the challenges faced in assessing content reuse through a comprehensive needs assessment of the digital library community. The eventual goal behind the grant is a multidimensional framework to support digital libraries in demonstrating their value and better advocating for the resources and platforms that will best serve their communities. In order to do this, a deeper understanding of how digital objects are used and repurposed is critical. This must be coupled with an understanding of what types of resources already exist, how reuse is or is not valued differently from access use, and what approaches and tools need to be developed. This paper shares the initial quantitative and qualitative analysis and results of the community needs assessment conducted in 2017 and 2018 that aims to illuminate digital library reuse strengths, weaknesses and community applications.

Literature review

In 2015, the white paper, “Surveying the Landscape: Use and Usability Assessment of Digital Libraries,” detailed both the efforts and the challenges in assessing content reuse (Chapman et al., 2015). Analyzing 26 articles published between 2010 and 2014, the authors focused on understanding current strengths and gaps in the research, with the aim of highlighting areas that could benefit from future study. This review found that there were promising assessment developments, including a growing body of research on reuse patterns and practices among content related to work in the humanities and arts. It also identified a series of deficiencies, including: difficulty in identifying who uses digital libraries (sometimes due to user privacy safeguards) and how research focus (e.g. general vs scholarly use) affects how and what is reused; a lack of research on methods to track online reuse through hyperlinking; and difficulties measuring the reuse of digital objects in virtual and analog environments. The white paper made several recommendations, including a call for additional studies to better understand diverse user groups, the connection between reuse and repository design, and when users can ethically and legally reuse digital library content. Most importantly for this study, the authors recommended “the development of a reuse assessment framework and an accompanying toolkit or best practices to help unify future studies and discussions of reuse in the digital library field.”

Since the release of “Surveying the Landscape,” more recent literature has addressed some of the identified research gaps, in particular, reuse assessment work done beyond humanities and arts user groups. One such area is data set reuse. Open access funding mandates, coupled with strong data management outreach efforts and a growing open science movement, appear to be facilitating reuse and its measurement in this arena. In the article “Discovery and Reuse of Open Datasets: An Exploratory Study,” the authors suggest that these efforts have led to digital data objects that are well documented, usable with different software, shared in open access repositories and using persistent identifiers, are allowing reuse to be both tracked and assessed (Mannheimer et al., 2016). Further, projects such as “Always Already Computational” start to bridge the gap between largely science research and the humanities user groups by positioning digital collections themselves as data sets (Padilla et al., 2016).

General movement within the digital library community helps in developing and broadly applying standards that demonstrate the impact of repositories abound. Although not specifically focused on reuse, two current projects should be noted. One, JISC’s Institutional Repository Usage Statistics (IRUS) UK[3] is working to aggregate and compare usage statistics across repositories using Counting Online Usage of NeTworked Electronic Resources (COUNTER) – compliant metrics[4]. Another, the Repository Analytics & Metrics Portal (RAMP)[5] is investigating the difficulties digital repositories face in tracking and evaluating use. Benchmarking at the aggregate has been problematic for institutional repositories, and these efforts highlight the current drive for adaptable, transparent metrics.

Rights have also risen to the surface, particularly research highlighting the relationship between copyright, permissions and reuse of digital materials. Clearer terms of use, the encouragement to embrace open access policies, and object attribution for digital content are critical to maximizing the potential for reuse (Terras, 2015). Unstandardized rights metadata can complicate users’ attempts to reuse content, such as that found in the Digital Public Library of America (DPLA), and this has accelerated efforts by such referatories to standardize aggregated rights metadata (Frick, 2016).

Recent literature has also highlighted some of the approaches information professionals can use to track and measure reuse over the web. Several studies have documented the benefits and limitations of using reverse image lookup to trace digital image reuse (Kelly, 2015; Reilly and Thompson, 2017). Kelly has also evaluated both the reuse of digital collection content in Wikipedia entries through tracking citations (Kelly, 2017a) and the employment of Google Alerts to signal image reuse (Kelly, 2017b). Further research has tested the viability of using embedded metadata to identify reuse (Bushey, 2013; Thompson and Reilly, 2018).

Fewer articles have tackled the challenge of formulating analytical frameworks for assessing reuse. In “Beyond Clicks, Likes, and Downloads: Identifying Meaningful Impacts for Digitized Ethnographic Archives,” the authors devised a framework for “documenting, demonstrating, and assessing the impact of digitized ethnographic collections” (Punzalan et al., 2017). The researchers formulated six topical areas of potential impact, including: knowledge, professional discourse, attitudes, institutional capacity, policy and relationships. They note that these areas can assist how “institutions and communities articulate and assess major sorts of impact that are most relevant to institutional projects to digitize and share knowledge.” This work shows the potential for information professionals to re-define the current framework for measuring impact. The research detailed here extends these kinds of analyses by collecting and analyzing community feedback to provide important use cases and functional requirements that expand the profession’s perspective on assessment.

Methodology

The structure for the needs assessment is both multi-part and iterative. It includes data collection and analysis of pre- and post-community surveys, in-person and virtual focus groups, and regular incorporation of advisory board and community feedback. The culmination of this work will be the identification of shared tools and best practices for reuse assessment within the digital library and cultural heritage communities.

The research team began the needs assessment by administering a pre-survey to digital library practitioners – its purpose was to identify potential gaps in reuse assessment practices. The survey instrument was developed on the Qualtrics platform, and included 19 questions aimed at gathering institutional demographic information, as well as statistics gathered for digital collection use, available infrastructure support, existing assessment barriers and what kinds of data would be useful for practitioners. The survey was distributed on 25 listservs related to research data professionals, cultural heritage organizations, digital library and library assessment groups. The Digital Library Federation advertised the survey as well on their social media channels. The full survey instrument can be found here https://osf.io/ptvh5/. Of the responses, 302 were kept for analysis. These were respondents that had agreed to the institutional review board statement and answered a minimum of one question.

The focus groups were structured after the pre-survey was completed. Their purpose was to gain qualitative insight into the use cases for assessment metrics, and to drive the investigations of the research team. Close attention was paid to gathering a broad perspective from academic, public and special focus institutions. Suggested participant criteria included: demonstrated experience building, preserving and/or assessing digital library collections; self-identified individuals from underrepresented groups; representatives from organizations with diverse and inclusive collections; and self-identified representatives from cultural heritage organizational types that might not be represented in the research team membership.

The focus group questions centered on identifying reuse assessment needs within different communities, tools and methods currently being applied, financial and institutional costs involved in reuse assessment, user privacy concerns, functionalities needed in assessment tools, and the costs of building and maintaining assessment instruments. For each focus group a facilitator and a note-taker attended the sessions. Prior to the session, participants received an outline of topics to be covered. Sessions were recorded to ensure all themes were captured. First round focus groups were held in October and December of 2017, participants came from academic libraries or archives (29 percent), public libraries (21 percent), museums (21 percent), consortiums (21 percent) and special focus institutions (8 percent). Participants included practitioners (58 percent), administrative decision makers (21 percent), as well as some in more removed roles at consortial levels (21 percent).

The second round of focus groups, held in February and March 2018, built off of the themes found in the Exploratory Focus Groups. Participants in this round were typically from academic institutions (80 percent) but there was also representation from public libraries, museums and special focus institutions, further detailed in the findings. Participant makeup was almost exactly split between practitioners and decision makers.

Researchers applied a grounded theory approach to analyzing data generated from the focus groups (Charmaz, 1983).The team used Dedoose to code the notes generated during in-person and virtual focus group sessions.

The full needs assessment is structured so that each component builds upon the completion of the last. For example, upon the completion of the pre-survey, the research team worked with the grant advisory board[6] to refine the approach to analyzing the results, and re-examine how to best structure future surveys. Likewise, the results of the first focus groups shifted the next focus group topics to areas that had warranted more discussion, including the definition of reuse, sticky metadata and privacy concerns.

This paper, and the methodology detailed, centers on the initial findings of the yearlong research project, including the results of the pre-survey, and preliminary analysis and noteworthy themes from two of the in-person focus groups and two of the virtual focus groups. Future analysis, research and subsequent steps are detailed below, but as with the approach thus far, future research will build upon these results. Next steps already informed by this approach include an upcoming post-survey, the results and analysis of an additional focus group, and the final recommendations for the functional requirements of a reuse toolkit.

Survey findings

Focus group findings

The following summaries are broken out by type of focus group (Exploratory or Technologies and Standards), and within those categories, by major topics. The project team conducted a total of three rounds of focus groups, with each round offering participants the opportunity to partake virtually or in-person. At the time of publication, only two types of focus groups (Exploratory and Technologies and Standards) had been completed, and that is the analysis summarized here. In total, the project team interacted with 38 people through these groups. Half of all participants (47.4 percent) represented academic libraries. Special focus libraries, such as corporate and government libraries, made up on nearly one-fifth of the participants (18.4 percent), with remaining participants representing museums (10.5 percent), public libraries (10.5 percent), consortia (7.9 percent) and academic archives (5.3 percent).

The project team conducted each focus group with two grant team members present – a facilitator and a note-taker. A set of questions was used to frame and guide the focus groups, and these were distributed to participants in advance of the conversation. Each session was recorded, and these recordings were used to supplement notes. Per the grant specification, audio recordings were destroyed after 48 h. The qualitative analysis software, Dedoose, was used to code the focus group notes for further analysis.

Conclusions

Analysis of the survey and focus group data highlights several key themes in reuse assessment. The most prominent of which is that the digital library community is looking for field-wide approaches for assessing the impact of reuse in order to better understand, and tell the story of, what has been learned or gained by a user when they repurpose a digital object. Although standardized assessment approaches are critical, it is equally important that they be both modular and flexible enough that a range of institution types can apply them within the context of community-specific values and needs.

Although a majority of institutions do track limited types of use data, very few track reuse data, and even less consistently do so across collections. For those that are tracking use data, analysis and dissemination of this information is haphazard. It is clear from this research that metrics developed must be purposeful, and link directly to the outcomes prioritized by the organization. Too often practitioners felt they were collecting data for data’s sake, and adding to that data noise would be a mistake.

Embedded metadata that is consistent across platforms will be important to tracking object identifiers and relational metadata. Ideally, content management or data collections systems can collect use and reuse data on a granular level, and institutions will have the ability to turn on or off the gathering of specific data points. An ability to interface with aggregator data is crucial for smaller institutions, and specific approaches to data collection and dissemination should be detailed in licensing negotiations with vendors. The IIIF, with its community research focus, defined APIs and compatible software, can serve as a model for the reuse assessment toolkit. Similarly, RAMP and JISC’s IRUS can serve as models for large scale aggregation and implementation of standards across institutional repositories.

Finally, this research points to the need within the digital library community for benchmarking, and the ability to show relationships and patterns among digital objects that are being missed by currently collected data. Although software and approaches have been identified, foundational assessment techniques and training opportunities may be more immediately useful to the community at large. This includes resources on everything from data visualization, to methods for translating data to impact statements, to building and normalizing assessment practices that are specific to cultural heritage, data and digital repositories, and digital library organizations.

Ethical considerations and community values should be at the forefront of all discussions, and a future toolkit should explore how data can be used, the good and the bad, to empower institutions to make decisions.

Finally, this research has made clear that the digital library community is hungry for ways to understand the users and uses of digital collections by applying a new type of assessment lens – a lens that examines how materials are reused and repurposed within various communities, rather than simply measuring volume of use.

Further study

Beyond the final analysis of the upcoming focus groups, post-survey, and framework development and recommendations, significant themes have emerged. Any year-long project that engages a large group of practitioners and interacts with a hands-on advisory board will inspire additional areas of study. During the course of the analysis to date, the following areas have emerged that warrant deeper interrogation outside the parameters of the current research project.

Re-contextualization concerns were surfaced by the focus groups in two ways. First, the ethical consequences related to the tension of capturing detailed reuse metrics with the challenge of third-party vendors using the information. In fact, this theme generated so much conversation that the project team added related questions to subsequent focus group outlines. Digital librarians expressed an interest in utilizing specific personal information within a digital library to inform priorities. Indeed, the potential for third-party vendors (such as Google) to use the same information in a context misaligned to the goals of the institution poses a threat to the information and its custodians. As a community keenly sensitive to the misuse (or perhaps this too is a form of reuse) of personal information, how should standards be set? What is the digital library community’s role, as a group that regularly interfaces with for-profit, third-party entities, in shaping this conversation?

Second, focus groups revealed that digital librarians prepare for, but may not feel empowered to handle instances where digital assets are being used in a controversial context. Re-contextualizing digital assets, especially for use by extremist, revisionist or hate groups likely occurs. Digital librarians rarely have a vehicle to consistently track usage of their images in any forum. However, in the situation where digital assets are being used to promote a credo that stands in opposition to that of the original collection or home institution, digital librarians and other stakeholders may find themselves in a difficult position. While legal deeds and rights may be cited to prevent institutional liability, digital librarians must consider the implications to the relationships forged, partnerships cultivated and personal histories misused.

Third, platforms need to keep up with practitioner needs and community usage of digital assets. Perhaps the only sustainable approach for digital librarians to interact with third-party systems that meet immediate (and changing) demands is to be involved in the development of new platforms and tools. Do digital librarians and information management professionals leverage positions with vendors who supply tools that are integral to daily operations in all the ways they can? Every focus group raised the topic of Google Analytics as a platform through which usage was collected and analyzed, which presents a portrait of their significance in the institution or the community. However, most focus group participants did not report using the same platform to track reuse. If reuse metrics are equally important to initial use, how can these types of tools assist?

Finally, the importance of shaping a strong narrative to demonstrate the value and impact of a digital library on relevant communities stands out as critical to the sustainability for digital libraries in a competitive environment. Stakeholders and funding sources in academic or cultural heritage settings can range considerably, and are quite diverse. One way this diversity impacts digital libraries is through their ability to tell stories to all the entities that need to hear them. Beyond use and reuse metrics, how can we empower the community to better convey the impact of the digital library and related services to stakeholders?

Through engagement with the digital library community the researchers initiated a series of conversations with experts where few were occurring around topics of reuse. As a result of this unearthing, the research team, along with the digital library community at large, has much still to do in an effort toward a fluency of the importance on reuse for the future of digital libraries.