Telling the story
Qualitative approaches to measuring the performance of emerging library services
The Authors
Peter Brophy, Centre for Research in Library & Information Management (CERLIM), Manchester Metropolitan University, Manchester, UK
Acknowledgements
© Peter Brophy, 2008
Abstract
Purpose – The purpose of this paper is to show that libraries are facing challenging environments in which services and content must be delivered into the users' workflows and “lifeflows”.
Design/methodology/approach – An understanding of linguistic codes and language games, and similar concepts, will help librarians to appreciate the enormous challenge of becoming an accepted member of each user community.
Findings – In order to assess performance in this changing environment, much greater emphasis will need to be placed on qualitative methods, including ethnographic approaches, externally-moderated, reflective self-evaluation and narrative-based practice.
Originality/value – Libraries themselves are changing, and most of all are being challenged to reinvent themselves within the workflows and lifeflows of their users. If this challenge is to be met then innovative yet robust methods need to be developed to assess performance in this new milieu.
Article Type:
Viewpoint
Keyword(s):
Qualitative methods; Linguistics; Information services; Language; Ethnography.
Journal:
Performance Measurement and Metrics
Volume:
9
Number:
1
Year:
2008
pp:
7-17
Copyright ©
Emerald Group Publishing Limited
ISSN:
1467-8047
Introduction
The question, “How good is this library service?” has a considerable history. We could go back to Alexandria, where comprehensiveness seems to have been the critical measure, or even further, but modern approaches to the problem started to come of age only in the late 1960s and 1970s. One particularly seminal contribution was made by Richard Orr (1973), who distinguished between two questions:
- “How good is this library?” where the issues include quality, effectiveness, efficiency, economy and, increasingly, equity.
- “How much good does this library do?” where the central concepts are those of impact, outcomes and value.
The first question is, in essence, internally focused and measures tend to relate to processes and outputs. The second is concerned with the effects the library has on its environment and, crucially, on its users.
Of course not all libraries are the same and it is very important that the particular mission, concerns and environment of each library is taken into account. Sectoral distinctions are important, but it is also noteworthy that societal pressures are resulting in changed understandings of libraries' roles. At a previous Performance Measurement conference, back in 2001, I suggested that these include:
- The invisible intermediary. Delivery of electronic content within users' workspaces, such as virtual learning environments (VLEs) and virtual research environments (VREs) may render the library contribution all but invisible to the user. Indeed this is almost inevitable if seamless integration is to be achieved.
- The memory institution. One of the traditional roles of the library is to preserve humankind's knowledge for the long term. The collected wisdom of the centuries must not be lost, yet in an age which so often seems to value the instantaneous and transitory over the measured and persistent, libraries have to work hard to justify the resources they expend in the interests of future generations.
- The learning centre, especially one understood as the place where people meet and exchange ideas, learning from each other and through the process of interaction. Here the relationship with more formal centres of learning, including schools, colleges and universities, is critical.
- The community resource, where the library provides a place – both real and virtual – for members of the community to meet and interact for a wide variety of purposes, including simple enjoyment. Increasingly this role needs to be seen as supporting people's creativity as well as supplying them with information.
- The traditional (but hybrid) library, with the full range of services which allow users to access the widest range of resources and to manipulate those of interest in a variety of ways, supported by advice and guidance from human intermediaries (Brophy, 2001).
What is central to all of these concepts, of course, is that they are centred on the user. It is therefore useful to spend a little time thinking about how the user experience is changing.
Library users
When considering how users and libraries interrelate there are of course an almost limitless number of possible scenarios. In this paper I want to concentrate on five, chosen because evidence suggests that they are among the most critical for understanding how we may need to manage user interactions in the future. These five scenarios examine the library user within: the networked landscape; the virtual learning environment; the research environment; scholarly publication; and, personal creativity.
The networked landscape
Many commentators have observed that the networked information landscape marks a revolutionary shift in user behaviour. The ability of users to access a vast range of resources from their own computers, without recourse to an intermediary, forces a rethink by those, like librarians, who previously had a near monopoly on access. We are now observing significant changes in library services as a result. Where in the past there would have been a queue at the reference desk, most users now rely on personal search for an answer to their queries, utilising keyword searches in Google or Wikipedia, or if they are a mite more sophisticated, perhaps Google Scholar or PubMed. In effect they have fingertip access to a global library. If their chosen source cannot respond in less than three seconds, or produces what appear to be meaningless results, they try elsewhere. As Lorcan Dempsey has characterised it:
Then: Resources scarce, attention abundant.
Now: Attention scarce, resources abundant (Dempsey, 2007).
Among the alternative sources which are utilised are what we might call “Phone a Friend” approaches: social networking sites, wikis and the like which enable problems to be discussed and solutions shared. The “friend” is no longer the librarian, but a colleague or fellow citizen known to share the same interests. Interestingly librarians do the same thing in their professional circles: the SCONUL VAMP work reported elsewhere in this conference is intending to continue its discussions through a wiki.
Another notable feature of the networked environment is that users are no longer satisfied by delivery of bibliographic and location data, or even find it acceptable as part of the solution. Instead they want the full text (or audio, or video or whatever). Discovery to delivery (d2d) is the watchword.
Old conceptions of recall and precision have largely been abandoned. In part this has occurred because they cannot be measured in any meaningful way, but it is also noticeable that many users are no longer persuaded by the idea of retrieving all the relevant information and only the relevant information. They simply want enough information to get by – they “satisfice”. To paraphrase Dempsey:
Then: what is the best source of authoritative information?
Now: which is the most readily accessible source of adequate information?
But what is perhaps the most notable feature of user behaviour in these environments is that the tools and resources they use are those which are embedded in their workflows. They do not want to switch applications or perform complex transformations of data. Neither, by and large, do they want to set aside a separate time for information searching and retrieval Information usage is integrated into the workflow.
The virtual learning environment
Many academic institutions have implemented virtual learning environments (VLEs) and are gradually making them more and more central to the delivery of their courses. They are not simply a vehicle for remote delivery of distance learning but a core tool for drawing together learning and teaching resources, including lectures, tutorials, seminars, discussions – and information. Tutors feed core resources (naturally, in full text) into their programmes, or provide lists of pointers to internet sources. All the student needs is in one place.
These systems are not optional extras. A student wishing to study at University X will have no option but to use University X's VLE. It is the defined workspace for the course. As this central role becomes embedded, tutors accept it as the home for course development, for course delivery, for course discussions and for course assessment. It is the home for learning workflows, what we might call “learnflows”.
Yet it is inhospitable to the university library, precisely because it is self-contained. To use library resources the student has to leave the VLE and enter the arcane world of library OPACs and the like. The only alternative is for the library to find ways to position itself in the learnflow. The former is simply unlikely to happen on the scale needed for the library to retain its position as a major player.
Research practice
One of the remarkable changes which has occurred in much academic research in the last decade is the extent to which experimentation and discovery has come to rely on the manipulation, analysis and characterisation of huge datasets. Obvious examples include astronomical research which relies on vast quantities of data streamed from satellites, nuclear physics with its reliance on particle accelerators, medicine with the analysis of the human and other genomes and geological exploration with seismic and other data. Very often scientists need to compare multiple observations, so that current datasets need to be handled alongside those previously archived. To achieve this, complex metadata is needed to ensure that the provenance of each set of data is fully understood. The need, therefore, is for curation of the data throughout its lifecycle.
The question for librarians, therefore, is whether they possess the skill set needed to be players within this environment. The need is not for a separate warehousing service which simply delivers up the data on demand. It is for the expertise to capture, store, describe, retrieve and process very large scale datasets which are in continuous use. So far there is something of a gap in the market, witness this comment from an astronomy research team, the Wide Field Astronomy Unit (WFAU) “(we have) recently recruited a dedicated science archive curator, who does not have an astronomical background. It is expected that the Unit will continue to require staff from a range of backgrounds, and with a range of skills: the technical requirements of WFAU's science archive curation greatly exceeds that which can be comfortably provided by professional astronomers” (Digital Curation Centre, 2005). Note that WFAU are not outsourcing the task to an external unit but bringing expertise in-house, where it can be applied within the workflow.
Scholarly publication
It is also informative to consider what is happening to scholarly publication. Again, this is an area seeing considerable change and it is difficult at present to predict the likely outcome of the battle between proponents of open access and those who would retain the traditional, publisher-dominated approach. However, among the straws in the wind are such indicators as the decision by all the UK Research Councils to mandate deposit in open access repositories of all publications based on research they have funded. It is excellent that academic libraries have been prominent in responding to this trend by setting up and administering eprint repositories in their institutions, often taking a lead to demonstrate the importance of this area. However, it is also noticeable that generally few academics have deposited significant numbers of their publications, although Markland (2006) remarks that “there is clear evidence at the present time of growth in the rate of deposit of documents of various kinds in institutional repositories”.
The reason for the slow development of repositories appears to be that the deposit of publications does not yet fit well within academics' workflows – it requires separate actions which have to be prioritised within busy schedules. Since the incentives are as yet unclear or at best unproven, it is not surprising that this activity fails to reach the top of the “to do” list!
Personal creativity
There is considerable evidence that users want to interact with libraries for rather more than information acquisition. The “discovery to delivery” (d2d) chain certainly describes the desirability of delivering full text without subjecting the user to time-consuming and sometimes irritating intermediary stages. But in many sectors it seems that what is needed is support beyond the stage of information acquisition into the users' own creativity. The eprint repository is, in a sense, a simple example of this, where the users utilises a library service to publish their work. Since they may also use library resources to spark their initial ideas, one might start to talk of a “creativity to creativity” (c2c) circle – a move from an information chain to a creative circle.
This enhanced role for the library can be seen quite clearly in the activities of public libraries. A recent study for the European Commission, the CALIMERA Action, examined innovative services being offered by public libraries in 42 European and neighbour countries (see www.calimera.org/default.aspx). This showed quite clearly that these institutions were providing a range of services to support their users' creativity, including learning. Services include the interpretation of resources (akin to a long-established role of museums), which spark ideas in users, to affirming and acknowledging individual user's creativity through exhibitions, including virtual displays of their work. “ICTs provide the opportunity for individuals and groups to become active contributors to the networked cultural environment. Tools are needed to enable and encourage ordinary people to express their creativity by telling stories, capturing images, recording verbal reminiscences and creating complex presentations using multiple media” (Brophy and Butters, 2007) Again, what is needed for this creativity to be expressed within the normal life experiences of individuals, rather than as something separated out which they have to seek purposefully. The workflows of life, or “lifeflows”, offer the environment for libraries to engage with their users in these ways.
Embedding services
It will be seen from the above examples that emerging service opportunities for libraries centre around the notion of embedding in lifeflows – whether these are in working, leisure, learning, personal health or any other environments. Library support needs to be surfaced where the users are, rather than offered as places for users to visit. This presents, of course, huge challenges.
As an aside it is worth pondering the question, “What is it that is distinctive about libraries in the modern age?”. Since users can and do look elsewhere for much of their information, have multiple sources of bibliographic information, can often find alternative study space and are offered advice and training from many different organisations, is there still a distinctive role for the library? An analysis of library functions suggests that there are two areas where libraries hold, if not a monopoly, then a distinctive advantage. These are firstly their collections, specifically those parts which are rare or unique and those which are subject to charges (and which individual users are unlikely to purchase for themselves). Secondly, they hold or have the potential to hold unique intelligence about their users. Other players, including global brands like Amazon and Google, make intensive use of their knowledge of their users' behaviour. For example, Google recently launched its ProfileRank algorithm to sit alongside the well-established PageRank: “the new Profile Rank is based upon user profiles built by tracking a user's web habits in and outside of Google Search, even if the user has not opted in to be served personalized results” (Baker, 2005). Few libraries have yet started to make use of this kind of information about their users. Few have apparently noticed the widespread use of customer relationship management software across many sectors. Perhaps it is time for that to change.
Engaging with users
It follows from the above discussion that libraries need to develop better understanding of their users and, more than that, to engage with them in their own environments. There are a number of theoretical perspectives which might help them to achieve this goal. In this paper I will refer to just two.
Linguistic codes
The concept of a linguistic code is widely used in human communication theory and elsewhere to express the idea that the language which any one group of people use demonstrates a different understanding of the world from that of other groups. It is simply not the case that each word in one language has an exact equivalent in another. Culler (1997), for example, points out that “speakers of English have ‘pets’ – a category to which nothing in French corresponds, although the French possess inordinate numbers of dogs and cats. English compels us to learn the sex of an infant so as to use the correct pronoun to talk about him or her (you can't call a baby ‘it’); our language implies that the sex is crucial … . But the linguistic marking of sex is in no way inevitable; all languages don't make sex the crucial feature of newborns”.
Clearly, linguistic codes apply not just to the languages used by people in different nations but to the way in which any particular language – English, French or any other – is used by specialists. Furthermore, understanding of the linguistic code comes about through use just as much as by language learning. As von Glasersfeld (1999) has put it, “children are not given a linguistic code in order to speak and understand language – they have to discover it on their own. They have to construct for themselves the meanings of a good many words before words can be used to expand the range of their linguistic communications.”
For librarians, especially those serving specialist groups such as academic researchers, it is just as important that the linguistic code is learned, understood and used. This brings us to a second, related perspective on this issue.
Language games
Ludwig Wittgenstein, the eminent philosopher, pointed out that the process by which we operate within a social setting is one he characterised as a “language game” (Wittgenstein, 1953). Just as a game has to have rules – though not all of them may be written down – so any social context has its rules, both explicit and implicit. It is important to be clear that when Wittgenstein talks of “games” he is referring, not to arcane and specialised rituals of communication, but to every kind of social interaction. In every social situation we learn the rules partly by being taught them formally and partly by observation. We see how other people act; we experiment with contributions of our own. The issue is not just about terminology but more critically about concepts and interpretation. It takes us beyond data, beyond information, and beyond knowledge towards shared meaning and shared action. In this sense it builds on the idea of linguistic codes to show how our every action is guided by our understanding of the “game”.
Working within academic teams is a classic example of language games in action. Gee put it this way: “An academic discipline … is not primarily content, in the sense of facts and principles. It is rather primarily a lived and historically changing set of distinctive social practices. It is in these practices that ‘content’ is generated, debated and transformed via certain distinctive ways of thinking, talking, valuing, acting and, often, writing and reading” (Gee, 2003). Librarians, to be effective, need to be fully part of the social practices of the groups they seek to serve.
Summary
I have written elsewhere that “what will underpin success in adapting to the new networked world will be a willingness and determination to meet users in their own environments, learning their terminologies and their languages” (Brophy, 2007). There is a huge challenge in those words. Librarians, if they are to remain credible, must develop a deep understanding of how the discipline or activity they are serving is practised. Only this will lead to credibility as a contributor to the enterprise and thus to the standing and peer regard to supply acceptable solutions to problems – preferably before those problems are perceived.
The key roles are already mapped out. They include subject librarians in academic institutions – as long as they spend their time with their user communities rather than holed up in the library; community librarians in public libraries – as long as they engage with what their potential users want to accomplish; and knowledge officers in workplace libraries – as long as they are willing to deal in personal know-how and not just information.
Measuring performance
What, then, does all this tell us about how we can measure the performance of a library? Clearly, traditional measures have their place – no-one would suggest that statistics of use are irrelevant, if only because the library which finds it is attracting no measurable use must surely be failing. However, if we are to take the need for embedded services seriously then we need to use methods which enable us to assess the contribution being made by the library to the broader aims of the organisation or community.
Looking to other disciplines, the most promising methods would seem to be those based on qualitative techniques. More specifically, we need a blend of three approaches:
- ethnographic methods, which rely on the assessor living in the lifeflow;
- independent, expert assessments; and
- painting rich pictures – in particular the use of narrative and story.
Ethnographic methods
Ethnography tries to take an holistic view of the world, recognising that human behaviour in its social setting is immensely complicated and that tacit knowledge and hidden assumptions – including linguistic codes and language games – are of great importance. The classic application of ethnography, epitomised by works such as Margaret Mead's Coming of Age in Samoa, involves becoming a participant observer within the society being studied, literally living within the society itself – what is known as cultural immersion. However, there are many variants of the original methodology, such as confessional and reflexive ethnologies, and the pure ethnographic concept of trying to become part of the group being studied is sometimes replaced with observation. In part this is a response to the criticism of ethnography that the participation of an external person from outside the culture inevitably affects the behaviour of members of the group being studied, although it is doubtful that this would be a major problem with, for example, an academic research group joined by a librarian.
Sarantakos (1998) suggests that ethnographic studies have four key characteristics:
- Culture. The focus of concern is on the shared culture of groups of people, examining and seeking to understand the shared patterns of behaviour, values, norms and standards.
- Holism. The focus is on the whole system being observed, rather than on an isolated part. Meaning and purpose can only be discerned within the context of the whole system.
- In-depth studies. Ethnography is not interested in the data collected by counting or even by questioning, but relies on “living in” the group that is being investigated.
- Chronology. Ethnography is interested in change, in how people and societies alter and reinvent themselves over time.
Two major types of ethnographic studies can be identified. Descriptive studies are usually called ethnographic. However, if a study is critical i.e. the data is used to try to identify cause and effect relationships, it is described as “ethnological”.
Independent assessment
Academic institutions are well-used to employing independent assessment, classically through external examiners and the peer review system in scholarly publication. For librarians these kinds of techniques have considerable value. They recognise that those responsible for delivering services are often too close to the service itself to make unbiased judgements. Using an independent reviewer provides balance as well as credibility to third parties, such as senior management.
A variety of techniques can be utilised. Experts may focus more on outcomes and impacts rather than the processes of engagement. Assisted independent assessment can be valuable, where the assessor is explicitly aided in engagement by the team under observation. A good example of this was the formal review of the UK's Digital Curation Centre where a process of externally-moderated, reflective self-evaluation was utilised. “The idea of this approach is that the DCC team, being closest to the activities which the DCC is undertaking and in a spirit of self-reflective practice, should undertake a series of actions to gather evidence of the extent to which the DCC is meeting its objectives. However, to ensure an appropriate level of independent assessment and objectivity, external moderators have been involved in the process, in terms of: advising on and agreeing the evaluation activities to be undertaken; involvement in those activities where appropriate; assessment of the evidence gathered; and the writing of the evaluation report.” (Brophy and Frey, 2006).
The use of narrative
In a number of disciplines, notably in health and in management, there has been a surge of interest in the use of narrative in evaluation, and indeed more generally. Stephen Denning, one of the “gurus” of this movement has written, “one reason why we live in a soup of narratives, why narratives permeate our lives and understanding, is that resorting to narratives is the way in which we have learned to cope with our world… Even while scientists and schoolteachers have been telling us to abandon these unscientific approaches, and adopt linear abstract thinking, the human race has used its common sense and stubbornly – to some extent surreptitiously – stuck with narratives as the most usable tool to cope with complexity” (Denning, 2001). In another contribution, he comments, “steadily increasing recognition of the importance of narrative in mainstream management is now inevitable … Narrative thinking is contributing to an emerging view of organizations that more accurately reflects not only the traditional structural, process-oriented, control-based aspects of an organization but also the living, flowing aspects of organizations – where talking, thinking, dreaming, feeling human beings work and play and talk and laugh and cry with each other, in a way that is organic and self-adjusting and naturally innovative.“ (Brown et al., 2005).
When we seek to measure the performance of our libraries, the challenge is to capture the rich pictures that retain the essence of our achievements without losing the richness of their contexts. Narratives are difficult to construct and equally difficult to use, for few people are natural story tellers. But if we can train professionals to analyse statistical data and present their findings in tables and graphs, surely it is worth the effort of training them in the art of storytelling. If we accept that access to our client groups can only be achieved by immersing ourselves in their language games and linguistic codes, there must surely be an equal imperative to communicate what is being achieved to the wider audience outside those walls. That is the contribution that narrative can provide.
It is immensely encouraging that the most recent Annual Report of the British Library has taken this approach in its online version. Its strapline is “The way we share knowledge is driven by our users. Discover our stories and tell us yours … ”. A typical contribution is from a user explaining why the BL is of such importance to him: “I'm Suvir Kaul, a visiting academic from the University of Pennsylvania and I've spent the last academic year here working on a book project on the impact of empire on the literary imagination in 18th century Britain. I have, for the most part, used the Rare Books Room and it has been an extraordinary experience, not simply because of the resources that are available here but because of the quality of the librarians who make those resources available to scholars. I cannot imagine a better place in which to work and I don't know of any comparable institutions anywhere in the world.” (British Library, 2007).
Conclusion
The measurement of library performance has evolved over the last thirty years at a significant pace and has helped inestimably in the improvement in the services which users receive. But we are at something of a watershed. Libraries themselves are changing, and most of all are being challenged to reinvent themselves within the workflows and lifeflows of their users. If this challenge is to be met then innovative yet robust methods need to be developed to assess performance in this new milieu. Because context is all important, and because of the complexity of information and communication flows in blended (real and virtual) environments, new measures much depict rich pictures of what is being achieved. Approaches like ethnography, externally-moderated, reflective self-evaluation and narrative-based practice will help library performance measurement to meet the challenges of the future and thus assist libraries themselves to retain and enhance their contribution to the wellbeing and advancement of our societies.
References
Baker, L. (2005), “Google Patent: organic results ranked by user profiling”, Search Engine Journal, 3 November, available at: www.searchenginejournal.com/google-patent-organic-results-ranked-by-user-profiling/2448/, .
British Library (2007), Annual Report and Accounts, British Library, London, available at: www.bl.uk/about/annual/2006to2007/index.html, .
Brophy, P. (2001), “Performance measures for 21st century libraries”, Proceedings of the Fourth Northumbria International Conference on Performance Measurement in Libraries and Information Services, Pittsburgh, PA, available at: www.libqual.org/documents/admin/brophy.pdf, .
Brophy, P. (2007), "Communicating the library: librarians and faculty in dialogue", Library Management, Vol. 28 No.8/9, pp.515-23.
Brophy, P., Frey, J. (2006), Digital Curation Centre: Externally Moderated Reflective Self-evaluation: Report, Digital Curation Centre, Edinburgh, available at: www.dcc.ac.uk/docs/DCC_Evaluation_Report_Final.pdf, .
Brophy, P., Butters, G. (2007), "Creating a research agenda for local libraries, archives and museums across Europe", New Review of Information Networking, Vol. 13 No.1, pp.3-21.
Brown, J.S., Groh, K., Prusak, L., Denning, S. (2005), Storytelling in Organizations: Why Storytelling is Transforming 21st Century Organizations and Management, Elsevier Butterworth-Heinemann, Oxford, .
Culler, J. (1997), Literary Theory: A Very Sort Introduction, Oxford University Press, Oxford, .
Dempsey, L. (2007), “The network reconfigures the catalog. (Slide 11)”, available at: www.slideshare.net/lisld/the-network-reconfigures-the-catalog, .
Denning, S. (2001), The Springboard: How Storytelling Ignites Action in Knowledge-era Organizations, Butterworth-Heinemann, Oxford, .
Digital Curation Centre (2005), Digital Curation Centre case studies and interviews, available at: www.dcc.ac.uk/resource/case-studies/wfau/case_study_wfau.pdf, .
Gee, J.P. (2003), What Video Games Have to Teach Us about Learning and Literacy, Palgrave Macmillan, New York, NY, cited in Tuominen, K., Savolainen, R. and Talja, S., “Information literacy as a sociotechnical practice”, Library Quarterly, Vol. 75 No. 3, pp. 329-45, .
Markland, M. (2006), "Institutional repositories in the UK: what can the Google user find there?", Journal of Librarianship and Information Science, Vol. 38 No.4, pp.221-8.
Orr, R.H. (1973), "Measuring the goodness of library services: a general framework for considering quantitative measures", Journal of Documentation, Vol. 29 No.3, pp.41-50.
Sarantakos, S. (1998), Social Research, 2nd ed., Macmillan, London, .
von Glasersfeld, E. (1999), "How do we mean? A constructivist sketch of semantics", Cybernetics and Human Knowing, Vol. 6 No.1, pp.9-16.
Wittgenstein, L. (1953), Philosophical Investigations, available at: http://users.rcn.com/rathbone/lw1-10c.htm, .
Corresponding author
Peter Brophy can be contacted at: p.brophy@mmu.ac.uk