An Action Plan for Outcomes Assessment in Your Library

Citation

Cullen, R. (2002), "An Action Plan for Outcomes Assessment in Your Library", Performance Measurement and Metrics, Vol. 3 No. 2, pp. 108-110. https://doi.org/10.1108/pmm.2002.3.2.108.1

New developments in the performance measurement of library and information services are extending our understanding of evaluation considerably beyond the spectrum of inputs/processes/outputs that have been the underlying paradigm for the past decades. These new developments include service quality measures and the balanced scorecard, already familiar to readers of this journal, and outcome and impact measures, the subject of the present review. They also include methods to assess the economic value and contribution to society of the activities and services of libraries (Morris et al., 2001; McEachern, 2001). All of these focus in some way on the fundamental question of the benefits that a community gains from investing in a library or information service, and challenge library managers to demonstrate the benefits of that investment. Outcome measures in particular are showing considerable promise of meeting growing demands for institutional accountability, especially in the tertiary sector. However, determining what those outcomes should be, and developing measures for them, whether these are the result of traditional or electronic services, has so far proved an intractable task.

Peter Hernon has tackled the task with his usual intellectual vigour and collaborated with co‐author Robert Dugan, Director of The Mildren F. Sawyer Library, Suffolk University, Boston, to produce a very practical manual which makes a valuable contribution to the emerging literature on outcomes measurement. Outcomes assessment, as defined here, is focused on local improvement rather than comparison with peer institutions; it asks how users have changed as a result of their contact with the library and its collections, and what the evidence for this change is. In the first part of the volume Hernon and Dugan focus primarily on academic libraries. In the tertiary sector in the United States, they note, “Federal and state government, the private sector, and the consumers of higher education (students and parents) want educational institutions to be more accountable for the funds, time and other resources allocated and expended in the education process” (Hernon and Dugan, 2002, p. 2). Libraries, as an integral function of institutions of tertiary education, are not exempt, and Hernon and Dugan cite a number of assessment criteria emanating from regional higher education accreditation agencies which list information services and information literacy skills gained by graduates as part of the institutional assessment programme. Chapters 1 and 2 outline some of the approaches taken by accrediting agencies, and individual universities and colleges, to assess the contribution made by the library to the educational mission of the institution, and the achievement of specific competencies in students. These competencies are most easily defined in terms of information literacy skills, and chapters 3 and 4 focus on information literacy, and ways of assessing outcomes in this key aspect of student learning.

While instruction in information literacy has considerable relevance to public libraries as well these days, as the authors readily acknowledge, their discussion of the measurement of outcomes in public libraries includes many other approaches. Chapter 5 outlines some of the performance measures commonly used by public libraries, making a clear distinction between inputs, outputs and outcomes, and notes that while standards are no longer the focus of performance measurement, especially for the Public Library Association and the American Library Association, measurement against agreed criteria on a population basis can produce useful measures of comparison. The chapter highlights the need for libraries to identify community needs and respond to these by adopting certain “roles” or service responses, as defined in the two PLA manuals Planning for Results (Hemmel and Wilson, 1998), and The New Planning for Results (Nelson, 2001). Outcomes assessment can be then be based on these.

Libraries contribute more than just information literacy training to the endeavours of students and faculty in tertiary institutions. Outcomes assessment must also include the impact of the library’s collections and on institutional outcomes themselves, or on individual or community wellbeing in the case of public libraries. Chapter 6 attempts to address this problem by focusing on how, once certain key roles are defined for either academic or public libraries, these can be used as the basis of outcomes measurement. In the tertiary sector, as we have already seen, these roles are primarily information literacy, academic performance, and life and career enhancement. In order to develop a coherent outcomes assessment programme, the authors take some of the rather vague statements made by the ARCL’s Taskforce on Library Outcomes, e.g. “By using the library do students improve their chances of having a successful career?” and “Are undergraduate students who used the library more likely to succeed in graduate school?”, which are full of unacknowledged confounding variables that could lead to biased and unhelpful responses, and show how these can be converted into more researchable questions, such as:

• •

  What do the students need to learn?

• •

  Is the library helping them learn it?

• •

  How is the library doing that?

• •

  How well is the library doing it?

• •

  How can the library sustain and improve its effort?

The authors suggest that these outcomes, with some modification, could also be relevant to public libraries and their communities of users. For those public libraries who are not happy with this approach, they also outline how the 13 service responses defined by the PLA’s New Planning for Results, e.g. consumer information, cultural awareness, government information, or lifelong learning, could be used to develop outcome measures that answer one of the key questions posed by this volume “How have users of our library changed as a direct result of their contact with our collections and services?”

Chapters 7 and 8 turn the reader’s attention to the use of research to provide evidence of the desired change in library users as a result of using the library’s services. The authors carefully outline a thorough approach to the development of a researchable question(s), beginning with the development of a problem statement as the foundation of reflective enquiry, and proceeding though the development of a logical structure, and definition of research objectives, leading to research questions and hypotheses, so critical to well‐designed research. Mastery of these basic research steps is essential if libraries are to collect and analyse data that will begin to answer the questions above. Chapter 8 follows with a useful discussion of a range of methods which could be used in outcomes assessment, once the basic research question has been defined. Focusing on both direct and indirect evidence, methods outlined include (in the first category): developmental portfolios; think aloud protocols (in which the activities of the user carrying out library based research are recorded and analysed); content analysis of research outputs and theses; pre‐ and post‐tests following information literacy instruction; surveys and interviews. Indirect methods include: focus groups; exit interviews; and external reviews in which the impact of the library can be evaluated. An example of how these might be carried out in practice and unified within one research framework is provided in a very useful Appendix supplied by Dugan’s Suffolk Library. The emphasis in this chapter is again on the academic library and the reader must go back to Chapter 6 to gain some insights into ways in which public library outcomes can be fitted into a similar framework.

In Chapter 9 the authors argue, with some success, that service quality and satisfaction measures should also be regarded as a form of outcomes assessment. Judiciously used to evaluate library services, and the broader learning and research activities of tertiary institutions, these can provide meaningful information about which services and activities students and faculty value. Although public libraries have not made extensive use of these methods to date they could find satisfaction and service quality useful measures in accounting for their outcomes. A concluding chapter stressing the importance of making a commitment to accountability and outcomes assessment helps set the agenda for the future: the need to improve on as yet imperfect assessment of learning outcomes; to develop a a new terminology and syntax, or what could be described as a new “discourse” for outcomes assessment, and the need for academic libraries to develop better relationships between libraries and faculty. Public libraries, the authors argue, need also to “develop partnership roles with public education, and other segments of the community, including the business community” in order to better assess their impact on their community.

Hernon and Dugan (2002) readily admit that the literature on outcomes assessment is only just forming, and that much of that literature is as yet outside the “the realm of library and information science”. The authors draw extensively on this literature to present their case, but do not entirely succeed in integrating the material into the fully developed overview suggested by the title. There is much more to be written and discussed about this topic before any kind of consensus is reached. The volume also has some shortcomings – with such a strong emphasis on educational libraries, the needs of public and special libraries are not well dealt with. While including satisfaction and service quality, and making brief mention of the balanced scorecard approach, techniques for assessing the economic value of library collections and services are not discussed and this is a major omission. However, a wide range of potential approaches to outcomes assessment are described in sufficient detail for libraries to make a start on this increasingly urgent task. The book makes a valuable contribution to what will be a challenging and vigorous debate over the next decade, and apart from a recent issue of the Journal of Academic Librarianship (2002), also edited by Hernon, which is focused on the topic, is the most comprehensive work published so far. As such it deserves serious consideration by everyone interested in the evaluation of library and information services, although not all will agree with its approach and conclusions.