Scholarly books and their evaluation context in the social sciences and humanities

Introduction

A famous novelist (Stephen King) once said: “good books do not give up all of their secrets all at once.” Knowing the author, this statement applies easily to horror and suspense fiction. Yet, it is fair to say that it is also applicable to “good books” published by scholars in the social sciences and humanities. Unraveling the secret to a scholarly book’s excellence, including its topicality, written delivery, influence, and longevity, clearly depends on time. Moreover, assessing what makes it “good” over “time” requires giving due consideration to the context in which the book was produced. In light of this, we have decided to open up a special issue in ASLIB Journal of Information Management concerning “Scholarly Books and Their Evaluation Context in the Social Sciences and Humanities.”

Initially, our aim was to elicit a variety of research contributions related to the following topics: scholarly book publishing, publisher prestige, quality and specialization, open access monographs, e-books, original language vs translated monographs, peer review standards and labels, commercial databases for books, national registries, books in social media and alternative metrics. This catalog is by no means exhaustive. Our impetus was simply to invite researchers to examine the special nature of books, which, thanks to Gutenberg’s press, have become widespread vehicles for sharing ideas and knowledge (Eisenstein, 2013). For centuries, books have thrived, and now, even in the midst of what was once “crisis” in the publishing industry, it would be an exaggeration to say that they are “dead” (Thompson, 2002). Calculating the number of citations books receive from journal articles and other books will neither help us process this industry threat, nor will the creation of new publishers necessarily motivate SSH scholars to produce more books. Likewise, current evaluation systems focused on the rapid production of research articles need not discourage book-oriented scholars either.

While scholarly books are typically excluded from national R&D reports, and considerations seem to be given too often to research papers indexed in databases like the Web of Science or Scopus, we encourage researchers and policy makers to push forward. Allowing scholarly books to do what they were designed to do – i.e. “give up all of their secrets” over time – lies with the facilitation of “quality” standards in refereeing, “quality” approaches to editorial publishing, and greater national and international record keeping. There has been some positive movement in this regard. In turn, we hope that the research community will continue to examine the visibility of scholarly books, and let stakeholders know when and where they resonate the most with scholars, as well as the general public.

National contexts, productivity and visibility

Amongst the list of topics presented with the call, there was one in particular that did not inspire new submissions; however, this was not surprising. Much of the earlier research concerning the evaluation of scholarly books came about due to the introduction of new commercial databases, like the Scopus Index for books and Clarivate’s Book Citation Index™ (e.g. Gorraiz et al., 2013; Kousha et al., 2011). In terms of “context,” what the research community expected was that these new indexes would represent the publication and citation interests of scholars internationally, but this has not necessarily been the case (Torres-Salinas et al., 2014b).

National book registries are one solution to the problem, and can give vital opportunities for cross-country comparisons, if not illustrate how culturally situated and language-specific book publishing is. This is perhaps the main reason why we received many article submissions that were focused on national, rather than international contexts. Thematically, this means that it was difficult to order the articles, since we do not believe that the issues and interests of one country supersede another. What was important to us was to find some way of presenting them, so that each reflects a logical, thought-provoking transition from one topic to the next. The list starts therefore with an overview of scholarly book publishing trends in Europe (in Flanders (Belgium), Finland, Norway, Poland and Slovenia), including insights into publication diversity (as in Poland), and country-specific policies leading to the scholarly book’s visibility or “invisibility” (as in the case of the Czech Republic).

In the first article, Engels et al. point to a variety of epistemic reasons as to why books are vital to the scholarly communication landscape. Articles from SSH fields generally do not supplant books; rather complement them, and it is well-understood that SSH scholars write books to facilitate the “explanatory” power of their work, in contrast to STEM (Science, Technology, Engineering and Medicine) fields, oriented toward incremental discovery. When this research team decided to assess the share of monographs and book chapters produced by five different European countries from 2004 to 2015 (Flanders, Finland, Norway, Poland and Slovenia), they chose to focus on both Economics and History, rationalizing that the two fields similarly value book publishing. Thus, it is the comparative cross-country patterns emerging from the study that proved to be interesting rather than the subject comparisons. With Poland in particular, we see a large drop after 2013 in the number of monographs published, particularly in the social sciences. Changes like this are often attributed to performance evaluation systems, which are often criticized for being detrimental. This might be slightly true for Poland, where relatively new reforms were put into place, but detrimental effects are not normative, especially when other countries with evaluation systems present stable monograph publishing patterns (e.g. Slovenia) or even a slight pattern of growth (e.g. Flanders).

With the second article written by Kulczycki, we obtain further insight into the situation in Poland, where the evaluation solution for books (notably the second cycle evaluation) is “distinct from other solutions used in European countries” because of its micro-level approach. In the absence of rated publisher lists (e.g. commonly used in Norway, Finland, Denmark), Poland has decided to put more focus on definitions of the book/monograph as well as other criteria (e.g. length, ISBN code: DOI) that need to be met for evaluation. Essentially, “a book published by Cambridge University press is counted in the same way as a book published by any small, local Polish publisher.” What stands out in this study is the fact that after 2013, when Polish language monographs were at their peak, SSH authors from this country began to write more often in different congress languages (e.g. French, German, Spanish) as well as in English. This partly explains the drop in monograph publications rates, seen in the previous study.

The Czech Republic is featured in the third article. Here, Broz and Stöckelová begin with a positive statement regarding the significant rise in SSH book publications registered by this country, citing the “gradual implementation of a performance-based system of research evaluation since 2004.” Read further and it becomes more apparent, from the ethnographic study of this sudden growth (i.e. interviews observations, policy document analysis), that what has happened within this country’s research community is not necessarily all positive. What the authors show is that the development of “in-house” publishing practices adopted by various Czech universities, and the use of foreign low-quality presses, has led to a “culture of orphaned books” and growing problem of “fake internationalism.” The “university press” in the Czech Republic has become what the authors refer to as author-managed self-publishing practices, which are not even occurring at the university level, but at the level of faculty/departmental units. The “orphan” metaphor that they use for scholarly books, establishes the fact that a large number never become visible, aside from being featured in registries or annual reports. In sum, “these books are hardly ever read, reviewed, purchased, or quoted.”

Performance-based systems: publisher lists and weighting schemes

The second two articles in this issue focus on performance-based systems, and the work that has been done to enhance evaluation procedures for books. In many cases, publisher lists have been adopted as well as weighting schemes. Usually, an editor evaluates a book before it is published; thus, an academic text never results in a finished “book” without the work of the editor. Publishers have thus become the focus of certain evaluation systems because judgments at this level support both the quality of the text as well as the added value of its editor. Policy makers responsible for creating publisher lists often pay attention to scholars’ perceptions (i.e. expert opinions) of what a prestigious publisher is in their field, but many choose to differentiate further between those with an international vs national reputation, in addition to those with and without rigorous external review procedures. With a publisher list in place, the basic role of a point-based scheme and potential weighting scheme for scholarly books, is to add value to it based on its publisher (i.e. prestigious or not; international or not) and the time it takes for the scholar to produce it, compared to a journal article.

Mañana Rodriguez and Pölönen present a comprehensive discussion surrounding the development of publisher lists, noting that one of the practical reasons behind their construction is that “the extensive, detailed reading of the contents of each [scholarly book] title under evaluation by a panel of expert is costly and time consuming.” Both authors are familiar with such lists, as they exist in their respective countries; however, in Spain, one is used to evaluate scholars at an individual level, while, in Finland, a publisher list supports the assignment of level points to books produced in research departments, later rewarded with funding. Their idea to test for and examine a “merging” of such lists is interesting, for two reasons. First, it verifies the extent to which they can be so different at a country level, given the obvious fact that each will include national publishers, but the authors point to added problems concerning the way that publishers may be ranked from a field-specific standpoint. Second, the study provides necessary insight for how to approach a larger, international motivation: to develop a useful international publisher registry.

Verleysen and Engels present another type of experiment, a “thought” experiment, concerning how weights are assigned to monographs, and here the focus is mainly on the Flemish performance-based research funding system (PRFS). First, they point to the uniqueness of the Flemish PRFS, which applies a weight to all peer-reviewed publications based only on publication type. A monograph produced by a Flemish author, regardless of where it is published, receives a point of 4. Likewise, a research article receives a point of 1. In contrast, other countries apply added quality levels to their weighting schemes (i.e. Denmark, Finland, Norway) so that, for example, a monograph published in a Nordic country with a “lower quality” national publisher receives a lower point than another published with a “higher quality” international publisher. Notable is the fact that each of the Nordic quality-weighting schemes vary. Referring to the various country-based schemas as practical, though somewhat “arbitrary,” the authors decided to test the adequacy of the Flemish 4:1 ratio, by proposing an “indicator of scholarly effort.” Weight is thus related to publication size, and operationalized at the level of disciplines and universities, utilizing a publication’s medium number of pages. What the indicator shows is that the augmented weight ratio at an aggregate university level does not have negative consequences for the funding system in Flanders. But, at the individual level, it suggests that the effort made to write and publish six pages of text for a peer-reviewed monograph would take the same effort as writing and publishing one page for a journal article.

Alternative approaches to book evaluation

With the last two contributions, we learn more about the educational value of scholarly books, both in terms of their teaching impact and public uptake via social media. University students can be enlightened by the explanatory power of a new SSH book as much as any reader outside scholarly communication system; however, little research thus far has given attention to this (e.g. Kousha and Thelwall, 2016; Torres-Salinas et al., 2017). Alternative approaches to book evaluation are also not included in the national performance evaluation systems featured in this issue. In terms of inclusion, we mean that they are not tied directly to funding or other incentives, but may be recognized informally, or under consideration for newer policies. With social media in particular, there are opportunities to assess a book’s broader impact, but as the last study shows, opportunities do not come without caveats.

Mas-Bledas and Thelwall point to the fact that scholarly books (i.e. edited with chapters and monographs) “do not have to follow the dense technical style typical of a journal”; and thus can “be more accessible to students.” In this paper, “accessibility” is measured in terms of the number of prestigious volumes added to teaching syllabi. In this sense, a “citation from a syllabus” becomes what the authors refer to as “an indicator of teaching value.” Publishers; however, are neither housed in every country, nor do they necessarily distribute books for use by students internationally. Hence, to test their assumption about “teaching value,” the authors focus on a sample of Spanish-language books (n=15,117) written and published in Spain. In addition to matching all Spanish-language books mentioned in syllabi, the study examines further the extent to which the same books have received citations in Microsoft Academic. Year by year correlation measures between total syllabus mentions and total citation counts yield positive values, though this study’s most interesting finding is that more books in the initial sample had received at least one syllabus mention than at least one citation in Microsoft Academic. Here, the books mentioned most often via online syllabi tended to be “monographs” from “the field of law.”

In the seventh, and final study of this special issue, Torres-Salinas, Gorraiz, and Robinson-Garcia take into consideration the role of Altmetric.com in the evaluation of books. Here, the authors’ concern is not so much “the insoluble problem of books” but the degree to which alternative research tools are effective for evaluating their visibility and impact. A fine-grained analysis of Altmetric.com is thus welcomed at this junction in book evaluation studies, since it is one of few resources that indicate where and when scholarly books are mentioned in social media. DOI’s are commonly used to trace journal articles, yet with books the opposite is true with ISBNs: “not all included in the Altmetric.com Book Collection have an ISBN code assigned to them.” However, the study shows that DOI’s attached to books are increasing, and while a total of “75% of books with a DOI do not show information related to their point of access,” a significant number lead back to Google books (books.google.com) and other significant resources. Perhaps the most critical note made by the authors is that there is a strong lack of documentation offered by Altmetric.com. Tests made with a sample of books extracted Clarivate’s Book Citation Index™ show that in the absence of “clear documentation” related to “input data and how books are identified,” the use of this tool can lead to a research process based on “trial and error.” This study points to additional issues that have yet to be studied (e.g. the inherent language bias of Altmetric.com), though with a positive outlook on what can be done in the future.