Editorial Peer Review: Its Strengths and Weaknesses

Jack Meadows (Department of Information Science, Loughborough University, UK)

Journal of Documentation

ISSN: 0022-0418

Article publication date: 1 February 2002

224

Keywords

Citation

Meadows, J. (2002), "Editorial Peer Review: Its Strengths and Weaknesses", Journal of Documentation, Vol. 58 No. 1, pp. 104-106. https://doi.org/10.1108/jd.2002.58.1.104.1

Publisher

:

Emerald Group Publishing Limited

Copyright © 2002, MCB UP Limited


Peer review is widely seen as an essential part – some people would say the essential part – of scholarly journal publishing. Each chapter of this book is headed by a quotation. At the head of the introductory chapter is the often quoted assertion by John Ziman: “The referee is the lynchpin about which the whole business of science is pivoted”. Because of the high level of importance attached to it, the peer review process in journals has been the subject of many studies over the past 50 years. Most researchers would expect Ziman’s assertion to have been well confirmed by now. Yet the quotation chosen to head the last chapter of the book is distinctly downbeat: “even at its best, the peer review process has inherent limitations”.

The extremely detailed survey of peer review studies presented in this book actually emphasises two things: not only that refereeing has its limitations, but also that studies of refereeing have theirs. Because of differences in methodology, analysis or population studied, it is often difficult to intercompare studies that ostensibly have the same objectives. This is true of investigations into most aspects of peer review. The author takes each major component of the process in turn – rejection rates, editors, authors, referees – and devotes a chapter to each, examining in considerable depth the studies that have been made. This leads at the end of each chapter to a summary of results, an identification of the main limitations, and recommendations for further investigation.

The literature search behind this book has been commendably thorough. Future work on peer review will certainly use the material as its stepping‐off point. Inevitably, there are some minor deficiencies. One is that a cut‐off date of 1997 was chosen (apart from a handful of later references to electronic journals). This is too early for a book published in 2001, and leads to the omission of some relevant material (e.g. McGinty, 1999). In some cases, summaries are quoted, rather than the fuller publication. For example, the article by H.‐D. Daniel (1993a) in Angewandte Chemie is cited, but not the more detailed discussion in his book Guardians of Science (Daniel, 1993b). This may have led to the omission of an interesting point relating to gender bias. As the author demonstrates, there is little evidence of a systematic refereeing bias against submissions from female researchers. However, the full ACLS survey (of which a summary is cited here) suggests that female researchers are more likely than male researchers to believe that bias of all kinds, not only against females, exists.

One obvious point that comes out of this survey is the subject bias present in peer review studies. The great majority to date have been carried out in the general areas of medicine, health and psychology. Such work as has been done in other fields makes it clear that results from the medical area cannot necessarily be generalised across the board. An entire chapter of the book is devoted to peer review studies that look at the use of statistical analysis in submitted articles. This is of vital importance in medicine, but of little interest in (say) physics or modern languages.

A slight bias in the book’s own survey is that the literature search concentrated primarily on articles containing empirical data. These undoubtedly contain the most important material. Yet some articles purely concerned with comment can suggest ideas for analysing the peer review process. Hypotheses integrating the whole field of peer review are underplayed here; and there is sometimes, perhaps, too much caution in trying to reach conclusions. For example, the question of cross‐disciplinary comparisons is treated very gingerly. Yet, as our own beloved research assessment exercise in British universities demonstrates, it is often necessary to make such comparisons in practice. In these circumstances, an informed guess is better than nothing at all.

As with any significant contribution, the book raises questions in the reader’s mind about things that are not mentioned, as well as those that are. For example, finance is only touched on lightly: just how much does it (or, rather, its absence) influence the review process? Or, again, how does monograph refereeing compare with journal refereeing? In passing, it may be observed that the British word “refereeing” has an advantage over the American word “reviewing” here. We can readily distinguish between refereeing the original manuscript and reviewing the published book, whereas Americans find the distinction more difficult.

In terms of what is provided, however, it is difficult to see how the contents could be improved. A necessary proviso is that this applies to studies of printed journals. Although a short chapter is devoted to peer review in an electronic environment, few studies of this have as yet been carried out. What future editions of this book may have to reveal is anybody’s guess. This apart, we have here an authoritative survey of the peer review process which should be read by everyone interested in scholarly journals. If the comments sometimes seem less than conclusive, this reflects as much the nature of the process itself as the quality of the studies carried out. The author’s final words provide a fair commentary on the situation: “Like a democracy, editorial peer review is messy and does not always work as it should, but it is essential to the integrity of scientific and scholarly communications”.

References

Daniel, H.‐D. (1993a), “An evaluation of the peer review process at Angewandte Chemie”, Angewandte Chemie, Vol. 32 No. 2, pp. 2348.

Daniel, H.‐D. (1993b), Guardians of Science: Fairness and Reliability of Peer Review, VCH, Weinheim, Cambridge.

McGinty, S. (1999), Gatekeepers of Knowledge: Journal Editors in the Sciences and Social Sciences, Bergin & Garvey, Westport, CN.

Related articles