Statistics, Measures and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Protocols

Everybody is doing it, but is everybody doing it well? The topic, of course, is digital reference, and as the lead author of this manual, Charles McClure noted in 2000 that there has been “considerable progress with development of digital reference services, [but] … very limited work to assess digital reference services”. This manual is an attempt to help fill the gap.

A manual consisting of over a hundred pages, mostly listing numeric performance indicators, might not be an easy read. The authors might have done well to make more prominent the final paragraphs of the section entitled “How to use this manual”, which recommends employing only a small number of the statistics offered. Once you realise this, you then have a way of tackling this manual – you read it with a view to select the measures of digital reference quality that apply most effectively to your own service. Moving through the text then becomes quite a stimulating process of discrimination, selection and rejection.

There are only two chapters in this manual, a short opener that sets the scene, followed by the bulk of the text in the chapter on digital reference statistics, measures and standards. These numeric indicators are grouped together in five categories: descriptive statistics and measures (including simple methods such as totting up incoming queries manually), machine collected statistics (such as web logs from a digital reference web page service), subjective feedback from users (focus groups, questionnaires), cost, and staff time spent. Each measure in the main five groups is helpfully described by means of a standard template – in each case you are given a definition (what the measure is), a rationale (why bother to collect it), a procedure (how to collect it) plus tools for data gathering, and a final discussion about issues and considerations. This discursive final section of template is the most readable bit, stimulating thought about whether to use a particular measure or not.

After these sections grouped around each of the five indicators, there are two further sections: a brief summary of other indicators that the digital reference team could not quite lick into usable form, and another oddly placed part of the manual right at the end of chapter two. This covers the issue of what you want to achieve in terms of digital reference service – courtesy, accuracy, cost and the like. Again, if placed more prominently earlier in the text this may have served as a way of laying down a marker. First you decide what quality standards you are aiming for, then you select indicators from the five main groups to give you data relevant to those standards, rather than the other way round. As the manual stands, you come to this section and wish you had read it earlier on.

The mistake you must not make in using this manual is to read it through thinking “we couldn't possibly collect that – it's too much work!”. There is no question of that, as the authors themselves indicate. The digital reference librarian must select, which core indicators are both relevant and practical for their service. The authors would have done this important issue a disservice if they had not enumerated every possible valuable measure and discussed each one in full. You may use very few of the manual's suggested statistical measures at the end of the day, but in exploring them all in the company of such intelligent authors your grasp of quality issues will be significantly improved.

So, this an important publication for reference librarians, not least for the interesting appendices (which includes a transcript of a model, high quality digital reference interaction) and for the useful references and web addresses of ancillary material as well.