The evolution of Global Libraries’ performance measurement and impact assessment systems

Purpose and background

The GL PMs are a set of indicators that measure the achievements and monitor the progress of GL’s Country Grant programs (The PMs closely follow the international standards in ISO 2789, 2006 and ISO 11620, 2008)[1]. Released in 2009, the initial version of the Global Libraries Guide to Impact Planning and Assessment contained a set of required and recommended PMs perceived as mainstream in the library field and desirable by managers, funders, and policy makers. In 2012-2013, after finding it difficult to aggregate and compare the data that had begun flowing in from several countries, GL consulted with grantee impact specialists to refine the metrics and to ensure that all grantees collect data in standard, comparable ways, using identical definitions and methods.

By providing a standard set of definitions and methods for use across countries, the PMs enable grantees to collect data that can be compared and aggregated for the purpose of collective learning, improvement, accountability, and advocacy. Today, all GL Country Grantees use these metrics.

PMs components

The PMs are grouped according to the following key categories:

• Public library service points metrics tell us how many libraries the GL program reaches:

  1. Public library service points include the total number of public library service points providing public access computing. A public library service point is any library facility, fixed or mobile, through which the public library provides a service to the general public. Central libraries, branch libraries, and mobile libraries are all each individual service points.

  2. Public access computing means providing at least one workstation available to the public regardless of whether access is free.

• Computers and workstations metrics tell us the amount of new technology available as result of the GL program. A workstation is a computer connected to the internet.

• Use of workstations metrics tell us whether library visitors are using the new technology the GL program provided.

• Visits to public libraries metrics tell us whether library use changes over time, particularly after the GL program provides new technology.

• Spending metrics tell us whether public investment in the country’s libraries changes over time.

• Training metrics tell us how many library staff and users receive training during the GL program.

• Library activities metrics tell us what library visitors are doing.

The following components accompany each metric:

• A specific definition for the metric.

• How to count: the sources, frequencies, and methods for data collection. These guidelines encourage proper sampling and estimating to avoid extensive measurement costs and level of effort.

• Required/Optional: if a metric is required, a grantee must collect the data. If the metric is optional, grantees may choose whether to collect this data.

• Target population: for some metrics, data should be collected from GL-supported libraries only. For other metrics, GL requests that data be collected (or estimated) for all libraries in the country.

See Appendix 1 for a complete list of the PMs.

The purpose

Today, the GL initiative seeks to equip public libraries with evidence of their ability to drive development – not just through traditional performance indicators, but in measurable results like job skills developed, education attained, employment found, money saved, and livelihoods improved. GL Country Grantees now employ a CIMS to quantify their work’s impact on public library users.

Data collected through CIMS enables public libraries to shift their focus from the services they provide to the outcomes they help individuals and communities realize. By agreeing to report the same measures using standardized definitions and methods, the GL initiative and Country Grantees are able to:

• aggregate data to determine the total impact of GL Country Grantees and enhance their ability to advocate for the importance of public libraries;

• track data over time to identify and monitor trends in public library use and reach, and incorporate this information into Country Grant programs and library services;

• compare data across countries to allow grantees to learn from one another’s successes and challenges;

• refer to a central, definitive source in communications and advocacy activities, so there is no confusion about where the numbers come from or how they are calculated; and

• leverage a new online reporting system, the Data Atlas, to visualize public library data, giving the GL initiative, and Country Grantees insight into dynamic results as they are reported.

The CIMS framework is a shared set of indicators that GL Country Grantees measure to understand their individual and collective impact on the lives of library users. These indicators – and the positive outcomes libraries hope to realize – span seven common issue areas (Table I).

See Appendix 2 for the complete list of required CIMS indicators across these seven categories.

Guiding principles behind CIMS

CIMS helps demonstrate progress toward a set of desired outcomes. For each of the seven issue areas, the GL initiative has identified a set of outcomes, or positive changes in the lives of individuals and communities, that libraries can help achieve. CIMS provides the indicators that help grantees collect data about whether these outcomes are occurring. Before the indicators were developed, a set of guiding principles were established to govern how the new measurement system would operate:

1. CIMS was designed with practical validity in mind, seeking a balance between what would be ideal to know and what is feasible to measure: each Country Grantee must feel comfortable that the data they collect paints an accurate picture of public library users in their country. To make the system manageable for Country Grantees with different levels of research experience and capacity, GL decided to work closely with Country Grantees and evaluation experts to develop the system.

2. CIMS emphasizes contribution, not attribution: CIMS is designed to help GL Country Grantees understand whether the efforts of grantees are one of the causes of improvements in the lives of library users – not whether (or how much) these efforts are directly or solely responsible. In practice, this means GL does not require grantees randomly assign their intervention to libraries and conduct experiments using control groups. In the contexts in which grantees work, choosing a random sample of libraries with which to work is often impossible (or extremely difficult). This sacrifice in the level of rigor is balanced by virtues such as practicality and grantee endorsement. Some grantees have chosen to compare data with findings from control groups, but this is not a requirement.

3. The CIMS framework includes both required and optional indicators: CIMS includes 41 required indicators for which all Country Grantees gather data (see Appendix 2 for a full list), and 53 optional indicators that Country Grantees may use if relevant to their program’s focus area. The GL initiative also encourages Country Grantees to collect additional, custom data to demonstrate the impact of their programs in areas of interest to specific stakeholders.

4. CIMS was designed to be manageable: the 41 required indicators translate into only 20 survey questions, a fraction of the size of surveys that Country Grantees already conduct. Grantees determine the format of the survey (paper or electronic), and the GL initiative recommends that grantees choose an external organization (like a polling firm or university) to collect the CIMS data (Figure 1).

The collaborative process used to develop CIMS

The CIMS outcomes, indicators, and methodology were designed in collaboration with the GL initiative’s staff, evaluation experts, and several members of each Country Grant team, including program directors and advocacy specialists – in all, over 50 people shaped the framework over a year-long process.

The GL initiative used this inclusive process for four reasons:

1. to promote a sense of shared ownership and familiarity with CIMS among Country Grantees;

2. to ensure that the indicators are truly relevant in the context of Country Grants;

3. to ensure that the methodology is practical and not a burden on Country Grantees; and

4. to leverage Country Grantee expertise in research, planning, evaluation, and indicator development.

Impact specialists – the Country Grant team members responsible for evaluation of projects and processes – were most deeply engaged in the design process. These grantees provided critical input on every step of the design of CIMS, from indicator selection to methodology guideline development.

To design the indicator system, facilitators used several types of processes to collect feedback. To generate long lists of potential indicators, the primary tools were:

• direct engagement: brainstorming exercises with grantee Impact Assessment Specialists and Advocacy Specialists at in-person meetings;

• individual phone calls with Impact Assessment Specialists; and

• group Skype calls with a cross-country pool of Impact Assessment Specialists.

After this initial set of exercises to brainstorm potential indicators for inclusion, facilitators led the participants through a set of activities designed to winnow down the list of 265 potential indicators into a manageable set that would not over-burden grantees. Techniques included:

• An online asynchronous voting exercise with all Impact Assessment Specialists, Advocacy Specialists, and Program Directors. Respondents were asked to vote on whether each indicator should be required, optional, or removed from the system entirely. During this phase Impact Assessment Specialists were encouraged to seek input from Country Grant team members, and to report back each group’s consensus votes. Each country provided consolidated results from multiple members of the country team.

• An online asynchronous voting exercise with GL staff and leadership, including 15 external stakeholders/consultants (e.g. from the Information School at the University of Washington).

• Facilitators color-coded the indicator votes and synthesized comments to help the GL team hone and prioritize the indicators.

Because 48 percent of the votes were for a proposed indicator to be “required,” GL worked to narrow the number of “required” indicators to a more manageable number of 41 required and 53 optional indicators.

Following a process of indicator refinement, survey questions were developed. The 41 required indicators can be collected through a mere 20 survey questions (11 content questions and nine common demographics questions) (Figure 2).

Overview of CIMS methodology

All grantees and external partners use the GL-provided methodological guidelines to collect the CIMS data using a Survey of Library Visitors and a Pop-Up Survey. These guidelines are provided to ensure that data can be aggregated and compared across countries and that field-accepted processes are employed. The guidelines also encourage proper sampling (and estimating, when necessary) to avoid extensive measurement costs and level of effort.

Question wording

Grantees are required to adopt the survey questions provided exactly as worded and then translate them into the appropriate local languages.

Translation procedure

The accurate translation of survey questions into local languages is integral to the success of CIMS. Therefore, grantees are required to use the following translation procedure:

• CIMS questions are translated by a professional translator;

• this translation is reviewed by the impact specialist for accuracy in the library context; and

• impact specialists adjust language where necessary to enhance accuracy.

In consultation with external data collection partners, grantees determine:

• the number of languages into which the survey should be translated (based on grantees’ understanding of the demographics in the regions where the survey will be administered); and

• whether and how much piloting of the translated questions to conduct.

Data collection instruments

Frequency of data collection

Grantees collect the CIMS data annually. They may schedule data collection to meet their needs and their reporting periods. One good practice is to align/integrate the CIMS data collection with existing plans to conduct impact assessment studies.

When a grant ends, GL considers providing post-grant support to help grantees or other institutions, such as governments or other partners, continue to collect the CIMS data. For example, in 2014 GL commissioned the research firm TNS Global to conduct a CIMS study in four countries where grant programs had ended: Bulgaria, Lithuania, Botswana, and Mexico. Conducting CIMS studies after grants have ended can provide insight into sustainability, offering powerful evidence that libraries are continuing to play an important role in their communities even after a large project ends.

Sampling

A great deal of time and effort went into creating sampling guidelines that achieve practical validity across countries by ensuring that survey responses reflect the diversity of library visitors. Grantees and external partners use the following sampling guidelines to select libraries where the surveys are administered and to determine the sample size for the surveys (Table II).

Comparing the use of technology at libraries across countries

The information in Figure 4 below focusses on several key digital inclusion CIMS metrics, displaying data by country, as well as a weighted cumulative average for each metric.

Many important data points are displayed in this figure, such as the success story that 57 percent of library visitors across all of the countries report that their use of technology increased as a result of public library services. Other country-specific insights are immediately available: for example, Ukraine has the second highest percentage of library visitors whose use of technology increased as a result of library services (81 percent) and the highest number of library internet users for whom the library is their only access to free internet (64 percent), emphasizing the importance of the role of libraries as a means to increase digital literacy of visitors.

These data also reveal that libraries are no longer simply a holding area for books, but are providing the tools to help patrons produce their own online content: 35 percent of library internet users across all of the countries created online content. Examples such as Latvia, where 77 percent of the library internet users generate online content, show that libraries are becoming places where local content is being created, not just consumed or checked out.

Gender case studies

Data Atlas users can also examine patterns across various demographic and socio-economic attributes of library visitors, including age, gender, race, urbanicity, education level, employment status, income, and disability. Of particular interest is that across this cohort of grantee countries, library users are predominantly female, often by a large margin. On average 58 percent of the library visitor population is female across the 12 countries that have reported CIMS data thus far (see Figure 5).

As many countries look to close the international digital divide, it is difficult to imagine much progress in a country’s information economy without the participation of women. However, according to Orbicom, the International Network of UNESCO Chairs in Communications (Huyer et al., 2005), women’s participation in the information society, particularly in poor countries, lags significantly behind that of men. And according to the Global Impact Study of Public Access to ICTs, conducted by researchers from the University of Washington (Sey et al., 2013), women accounted for just 29 percent of unique visitors to public internet access points such as libraries, cyber cafes, and telecenters in the countries they studied (Brazil, Chile, Ghana, Bangladesh, and the Philippines). (In that study, libraries had the highest proportion of female visitors, at 47 percent of users, compared to 28 percent for cybercafés and 23 percent for telecenters).

Findings from the CIMS data available in the GL Data Atlas are encouraging and reveal that libraries are emerging as an important public access venue for closing the gender digital divide.

A deeper look at Indonesia

According to the Center for Science and Technology Development Studies at the Indonesian Institute of Sciences, only 35 percent of the internet users in Indonesia were female in 2010 (Hermawati and Saari, 2011). Interestingly, the GL Atlas reveals that something different and special is happening at public libraries, with survey results from participating Indonesian libraries showing that 62 percent of library internet users in Indonesia are female (see Figure 6).

While further validation is needed to determine the effects of sample weighting and other factors contributing to these statistics, these initial findings tell a powerful advocacy story that libraries appear to be a positive force for increasing women’s online participation around the world.

The Data Atlas can also reveal findings that can be useful for program design. For example, in contrast to a female majority of internet users in other countries in the GL cohort, the Data Atlas reveals the opposite gender gap in Botswana, with women representing 44 percent of library visitors and only 33 percent of library internet users (Figure 7). Insights like these can hopefully help raise awareness and help grantees, as well as national, regional, and local libraries regarding their outreach and programming.

Employment case study

One critical opportunity offered by public internet access at public libraries is providing access to employment services. The dashboard below displays information about how library users search and apply for job listings at libraries in GL cohort countries (Figure 8). On average, 12 percent of library internet users search for job listings, of which 8 percent go on to apply for a job. On average, 23 percent of these applicants report receiving a job offer after using the library to apply for a job, a powerful example of how libraries can contribute to strong employment and economic growth.

These examples show a small portion of the data made available through the GL Data Atlas, revealing a glimpse of a rich and robust repository of research findings and powerful statistics for advocacy. The prototype web site remains in development, with the objective that it exist for years to come as a powerful tool to help analysts derive meaning from large library – and ICT-related data sets in support of sustaining library impact around the world.

Adapting PMs and CIMS in resource-constained environments: a case study from IREX’s “beyond access” project

With significant resources and funding from the Foundation, GL’s Country Grant programs have demonstrated the range of benefits that are achieved when libraries are modernized and better aligned with community needs and interests. But large investments at scale, often with a strong partner such as a National Library or a Ministry of Culture, are not always feasible. IREX’s Beyond Access initiative was conceived in 2011 as a model that would consolidate lessons from the Country Grants and opportunistically integrate that approach into work that governments and international development organizations were already doing. Beyond Access set out to build momentum for library-based collaborations with existing development initiatives.

Through advocacy and engagement activities, Beyond Access recognized that there is an opportunity to tie libraries to development quickly and sustainably. Since that time, and without country-based staff, Beyond Access has designed and implemented projects in several countries, including Myanmar, the Philippines, Peru, and Georgia.

Beyond Access projects do not typically support comprehensive evaluation resources in country. Each project operates as a coalition of local organizations including libraries, government agencies, and NGOs. As a result, the program relies on a local partner (usually a local NGO) to provide in-country support for both project implementation and evaluation. While this approach can increase sustainability prospects, these partners are unlikely to have dedicated evaluation staff, and many have limited experience in monitoring and evaluation. The most common funding resources for these partners are sub-awards under international funding mechanisms or through local sources. These funding streams are unlikely to include sustainable funding for evaluation.

As a result, when the PMs and CIMS were established for the Country Grants, Beyond Access was faced with a challenge: how could the comprehensive list of 63 PM and CIMS measures be distilled into a list of essential indicators that would not be burdensome for implementing partners to track? Could an approach be developed that would be useful beyond the life of grant funding? In the end, Beyond Access settled on 16 indicators (shown in Table III) that are tracked across all projects. These indicators were selected because Beyond Access believed they represented reasonable overlap with what libraries already tracked, while pointing toward new ways to articulate data in a way that would resonate with development audiences.

These core indicators draw heavily from the GL PMs (indicators 1-5) with additional indicators that capture the integral role of partnerships at local, national, and global level in the Beyond Access model (indicators 8-16).

The PMP incorporates a single indicator (indicator 7) that addresses the seven CIMS impact domains. This indicator serves as a meta-indicator to capture impact in a variety of development domains across all of the Beyond Access projects.

At the outset of each project, Beyond Access staff work with the implementing partners to identify project level indicators and to train them on monitoring and reporting requirements. The primary goal during this process is to prioritize the data that libraries can easily collect in their libraries and leverage for advocacy, so that the data collection and use is sustainable beyond the life of the project and partners appreciate the value of these data. This starts by identifying data that are already collected in project libraries (e.g. number of library visits) and streamlining how that is reported to the implementing partner (usually via an online survey).

CIMS can then be incorporated at an individual project level. Beyond Access works with each project’s implementing partner to identify targets that align with advocacy goals. These targets are typically output indicators that align with CIMS development domains, such as the number of people that receive basic ICT training (Digital inclusion) or the number of people that receive workforce development training (Economic development). These data are supplemented by a user survey that is structured so that it can easily be administered by library staff. The survey questions can be derived from CIMS survey questions that align with the project’s target domains and advocacy objectives.

Limited resources and partner capacity are real constraints that Beyond Access grapples with, yet by simplifying the indicators and providing training and support at the outset to implementing partners, Beyond Access has designed an M & E approach that allows for essential key learnings to be identified and shared.

For more information about the Beyond Access initiative, see www.beyondaccess.net