Metrics and epistemic injustice

Abstract

Purpose

This paper examines the socio-political affordances of metrics in research evaluation and the consequences of epistemic injustice in research practices and recorded knowledge.

Design/methodology/approach

First, the use of metrics is examined as a mechanism that promotes competition and social acceleration. Second, it is argued that the use of metrics in a competitive research culture reproduces systemic inequalities and leads to epistemic injustice. The conceptual analysis draws on works of Hartmut Rosa and Miranda Fricker, amongst others.

Findings

The use of metrics is largely driven by competition such as university rankings and league tables. Not only that metrics are not designed to enrich academic and research culture, they also suppress the visibility and credibility of works by minorities. As such, metrics perpetuate epistemic injustice in knowledge practices; at the same time, the reliability of metrics for bibliometric and scientometric studies is put into question.

Social implications

As metrics leverage who can speak and who will be heard, epistemic injustice is reflected in recorded knowledge and what we consider to be information.

Originality/value

This paper contributes to the discussion of metrics beyond bibliometric studies and research evaluation. It argues that metrics-induced competition is antithetical to equality and diversity in research practices.

Citation

Ma, L. (2022), "Metrics and epistemic injustice", Journal of Documentation, Vol. 78 No. 7, pp. 392-404. https://doi.org/10.1108/JD-12-2021-0240

Introduction

Publication- and citation-based metrics (hereafter “metrics”) [1] are not good or bad in-and-of themselves. In bibliometric and scientometric studies, they are numbers that can shed light on authorship patterns, research growth, the structure of scientific collaborations, as well as gender and ethnic disparity in research productivity and performance. In research evaluation, metrics are used in the decision-making process of recruitment, tenure, promotion and funding (Langfeldt et al., 2020). Despite the problems of citation analysis (MacRoberts and MacRoberts, 1996), it is generally agreed that metrics are reliable at the aggregate level (Aksnes et al., 2019). The overwhelming significance of metrics in research evaluation, however can compromise the validity of citation and publication data: when metrics are gamed and manipulated, they are less dependable as indicators of research quality and impact. Notwithstanding metrics are devised to be neutral and objective indicators, the use and misuse of metrics in research evaluation can jeopardise the validity of bibliometric and scientometric studies.

Problems and issues associated with the use of metrics have raised concerns about research culture and research integrity. Weingart (2005) warns that “the academic culture in which knowledge production thrived on a unique combination of competition, mutual trust and collegial critique is being destroyed” (p. 128). The edited volume, Gaming the Metrics (Biagioli and Lippman, 2020), documents misconduct and malpractice in research practices including citation cartels and fraudulent research. The Wellcome Trust (2020) Report on research culture shows that “43% [of the respondents] agreed that their institution/workplace placed more value on meeting metrics that it does on research quality, and 58% disagreed that the current metrics had had a positive impact on research culture”. Following the publications of the Metrics Tide (Wilsdon et al., 2015), the Leiden Manifesto (Hicks et al., 2015) and the Hong Kong Principles (Moher et al., 2020), more institutions have become signatories of DORA (San Francisco Declaration on Research Assessment) and some have published their own statement of responsible metrics [2]. These statements carry a very similar message: evaluative metrics should be used responsibly, specifically, metrics should not replace expert judgments when assessing individuals' research performance. These statements do not explicitly state the problems and issues pertaining to the use of metrics in research evaluation, nor do they indicate the irresponsible acts, while implying that the use of metrics is the cause of strategic behaviour, malpractices and fraudulent research. Yet, they do not object to the use of metrics and indeed support the development of better metrics. For example, one of DORA's recommendations for publishers is to “[M]ake available a range of article-level metrics to encourage a shift toward assessment based on the scientific content of an article rather than publication metrics of the journal in which it was published.”

Citation analysis was first devised to assist librarians in collection development (Garfield, 1967). Why and how do they become so prominent in the everyday life of researchers, research managers, universities and funding agencies? What is the allure of metrics? To understand the trust in numbers, many have argued that quantitative measures have been devised as a technocratic apparatus simply because numerical evidence is seemingly factual and difficult to dispute (Porter, 1995; Power, 2004; Mau, 2019). The allure of numbers is in its power to simplify complicated and complex processes without the need for lengthy debates and deliberations; the comparisons and rankings generated using quantitative measures grant authoritativeness in guiding actions and decisions (Espeland and Sauder, 2007). While metrics are not good or bad in and of themselves, the use of metrics can be claimed as the reasons for changes in research and publication practices. In the context of research evaluation, metrics such as Journal Impact Factor (JIF) and h-index become the gold standard that replace expert judgement in the assessment of quality and impact of research. The status of metrics is established first by non-reflective, ritual use of peer assessments, which is then taken up by other stakeholders who more than often have little knowledge of what metrics represent, how they are generated and what they can be used for. As metrics are established as the arbitrator of winners and losers in the competitive academic marketplace, research and publication practices evolve accordingly, to the extent that gaming strategies, malpractices and misconduct are frequent and noticeable.

Metrics have been symbolised as a culprit in the discourse of responsible metrics, yet it is important not to presume a causal relationship between metrics and undesirable outcomes. We must ask: Why does the use of metrics lead to malpractices and misconduct? Is it because of the very existence of metrics? Is it only because decision makers are using metrics irresponsibly? Or, are there other factors and mechanisms in play? When the academic job market and funding opportunities are often described as hypercompetitive and when many are concerned about the neoliberalisation of universities and research institutions (e.g. Hall, 2016; Olssen, 2016; Slaughter and Rhoades, 2004; Stephan, 2012), it is necessary to examine the socio-political affordances that drive the use of metrics in the system of knowledge production. Following Ma's (2021a) argument that metrics engender competition between researchers which in turn deprive them of time [3], I argue that the use of metrics is a symptom of the competitive and meritocratic research culture and that ill-considered competition can lead to deleterious consequences and exacerbate systemic inequalities and epistemic injustice and, consequently, what will be published and recorded as knowledge/information.

The paper is structured as follows: in the section, “Metrics for competition”, I examine the use of metrics as a technocratic device that promotes competition and social acceleration, perpetuated by university rankings and league tables. In the next section, “In Pursuit of Inequality”, I propose that metrics-driven competition is antithetical to the ideals of equality and diversity in knowledge practices. Then, in “Epistemic Injustice and Information”, I argue that metrics as a steering medium in the system of knowledge production leads to testimonial injustice—the devaluing of knowledge claims by prejudicial stereotypes. In the concluding remarks, I contemplate who is responsible for responsible metrics.

Metrics for competition

The chase after metrics—more publications, higher impact—is a symptom of the hypercompetitive academic marketplace. The low chance of survival or success compels researchers to work more and harder, while some even resort to producing research articles using fabricated data. Describing the unsustainable growth, hyper-competition and worth in life sciences, Fochler et al. (2016) report that “individual's worth is defined by their ability to succeed in competition, based on their productivity in acquiring tokens of academic quality that may count for something in the international academic labor market, that is, indexed publications, grant money and recorded citations” (p. 190). The report, What Researchers Think About the Culture They Work In (Wellcome Trust, 2020) states: “They [The researchers] accept competition as a necessary part of working in research, but think that it is often becoming aggressive and harmful. They also have widespread concerns about job security – especially in academia” when 78% of the respondents agreed that high levels of competition had created unkind and aggressive research conditions. At the same time, manager-academics in UK universities are experiencing “survival anxiety” that audit—and the use of metrics—becomes a type of defence mechanism (Loveday, 2021). Senior academics are primed to secure their positions by acquiring more funding and producing more publications, while early-career researchers are struggling to find tenure-track or permanent positions. The competitiveness means that researchers are increasing the amount of work per minute/hour/day and keep working when they are supposedly “off”. Metrics are used as a technocratic apparatus that enhances productivity by making researchers contribute longer hours and more days. For some, to the extent that one gives up physical and mental health, family life and friendship, all for the dream of an academic position or grant that is seemingly an honour and hard-earned merit.

There is no question that “competitive” is the current state of academia. The logic of competition is manifested in lists and tables—university rankings, annual highly cited researchers list, the 10 most beautiful campuses in the world. For universities, there seems to be no escape from competing for higher positions in university rankings as a response to the necessity of public accountability and transparency. Globalisation of higher education has led to the demand for international students, which in turn perpetuates the competition for “excellence” (see Hazelkorn, 2011; Hertig, 2016; Moore et al., 2017). Interestingly, however, research is considered more significant than teaching in university rankings such as Academic Ranking of World Universities (ARWU, commonly known as the Shanghai Ranking), Times Higher Education (THE) Ranking and QS Ranking—all of which include publication or citation metrics in their methodologies. These rankings prompt the chase after publications, impact and international collaborations which can be traced and indicated by metrics despite the different coverage and limitations of various databases. The chase after metrics at an institution level is initiated by the “battle for excellence” (Hazelkorn, 2011), the capacity “to compete as a world-class university” (Hertig, 2016) as metrics are seen as the major instrument to gain or maintain a university's place. When rankings become a priority or a necessity, the competition becomes heated by “repeated public comparisons of performance” (Brankovic et al., 2018). Although some may be critical of metrics and university rankings, they are obliged to engage in the competition and the pressure trickles down. In some cases, researchers are only represented and recognised by numbers on a spreadsheet.

League tables, rankings and lists are technology of competition where we can see the winners and the losers. Quantitative indicators—metrics—are used to support the results: journals can be ranked by Citescore or similar, publications can be ranked by the number of citations, researchers can be ranked by the number of publications or citations or h-index and universities can be ranked by a composite score that presumably represent how good they are. Without quantitative indicators, we could be arguing who is the best for forever. Once rankings and shortlists are created, it is as if there is no doubt that the results are based on fair comparison. Their methodology and data sources are seldom questioned. Quantitative indicators are more trustworthy than expert opinions. In trusting league tables and rankings, the scholarly community has assigned moral and political authority to those who create the rankings and shortlists and to those who produce the metrics.

Contemplating the relationship between acceleration and alienation in contemporary society, Hartmut Rosa (2010) maintains that the logic of competition is the main driving force of social acceleration, for which achievement is the determining or discriminating principle. Consequently, the more competitive a society is, the more accelerated it becomes. Universities and research institutions are not outside of these logics. The call for slow scholarship, for example, reflects the acceleration of knowledge production and what it demands of researchers and scholars. The cycle of work has been described as hamster-wheel that is non-stop and unceasing; and there is the sensation that if one does not catch up with the speed of the hamster-wheel, one is bound to be outpaced and hence out of the game. There is no longer a norm regarding an acceptable number of publications, now most strive to publish as many as possible due to the uncertainty of what would be enough in comparison to others, to not lag behind. It is as if researchers are forced to participate in this competition where only those who can churn out more publications in a limited time will survive or “win”. There is not only competition, but also acceleration in the ways we work.

For Rosa (2010), the logic of acceleration is totalitarian when “all individuals and political energies being sacrificed to the acceleration-machine” (p. 82). In the context of knowledge production, metrics are embedded in the larger system of knowledge production, researchers and universities cannot escape from the constant scrutiny by the metrics scoreboards on SciVal and university rankings. Felt (2017) has described the politics of time governing academic knowledge generation, epistemic entities and academic lives and careers as “chronopolitics”. Notwithstanding the warnings about the negative consequences of metrics-based research evaluation, it is seemingly an impossible battle to be rid of metrics or competition. In fact, there have been more metrics for comparison and more rankings [4], for a chance to win. Davies and Petersen (2005) show the inescapability of competition and acceleration by transforming an interview transcript into a poetic expression:

You do not have time to actually stop and think

And you often do not have time to speak about it

but I do think it happens in a very insidious way –

and I do try to resist it – hugely

But I think it is happening all the time

Surveillance at the heart of things

Reinforcing a police state; fascism, conformity

That amounts to a huge sea change in what it all means

while we do not even realise that it's happening –

That's what's so scary about it

And the university is just part of the technology for managing a

police state

Competition puts researchers against each other—because going up the rankings means that one must be “better”. Metrics make us easily comparable and put us into social hierarchies, and because the competition is continuous, the standard for excellence is constantly moving. When merit is celebrated in academic culture, one's failure to achieve is one's own fault. But as Michael Sandel (2020) explains in his book, The Tyranny of Meritocracy, meritocracy has its dark sides: that public discourse can be hollowed out and that systemic and structural inequalities are not addressed properly.

In pursuit of inequality

Meritocracy traps elites in an all-encompassing, never-ending struggle. Every colleague is a competitor. At every stage, the alternative to victory is elimination. – Daniel Markovits (2019), The Meritocracy Trap.

In competitive sports, the team or individual who scores more or runs faster or jumps farther is the winner. Although there are times when some would comment that Letesenbet Gidey is a better runner than Sifan Hassan, Hassan nevertheless won the 10,000 metres race in the Tokyo Olympics. There is no question about the result. But can we say the same for scholarly work? Is a publication that receives 300 citations better than the one that receives 298 citations? The answer is probably a resounding no. The reason is that when we think about a race or a soccer match, the track or the playing field is bounded, in the sense that one competes on the same ground rules and conditions at the same time and place. The criteria to win is clear and transparent, notwithstanding one can quip about how the other team or person actually played better or how the referee was somewhat biased. For athletes, doping is a fraud; a false start on a race and one is disqualified.

The pursuit of metrics in research practices—be they for benchmarking or mere survival—is not the same as a competition in sport. First, it is because the scoreboard of metrics presents false comparisons of a myriad of research areas, topics, specialisations, styles and intended audience. Although the scoreboard of metrics is meaningless when the publications and their authors and affiliations are not in the same “race” or “game”, they put researchers, departments, universities and even countries in competition as if these numbers are comparable. It is as if the scores of a football match can be compared to a rugby game. Second, the pursuit of metrics is a continuous effort that takes place in different locations. These locations such as universities and research centres afford researchers with different levels of resources and support. Lists of publications and researchers ordered by metrics are not the same as the results of a marathon because individuals can build on cumulative advantages while some are disadvantaged by, for example, high administrative and teaching load. More importantly, these lists neglect the manifold factors that constitute barriers to success such as factors that lead to disparity in productivity and performance including implicit or unconscious biases based on gender and race. Third, there are no clear rules and regulations in the pursuit of metrics. Unlike an athlete, a researcher who aspires to top the chart of metrics by strategizing publication practices or bullying junior colleagues are seldom called out or punished. In fact, their “success” may even be celebrated. Mostly, misconduct and malpractices do not exclude one from competing in the pursuit of metrics.

Not only that metrics rank and create hierarchies often without regard to epistemic cultures and norms, their functions as a steering medium in the system of knowledge production can also displace consensus formation in academic discourse (Ma, 2021b). The ethos of science (Merton, 1973, pp. 271–278; originally published in 1942)—universalism, communism, disinterestedness and organised scepticism are side-lined, if not replaced by “a scientific meritocracy” (Reinhart and Schendzielorz, 2020; see also Mijs and Savage, 2020) because of the highly skewed distribution of recognition. As the strategic competitors seek to interpret the “rules” of the competition to their advantages, metrics do not level the playing field. Contrarily, as Davies (2014) aptly points out, “devices of measurement and evaluation become tools of violence, to be used by one agent against another, rather than as a basis for consensus-formation” (p. 161).

Many have also argued that neoliberalism and new public management as the driving forces that compels the use of quantitative indicators with an expectation that there can be infinite growth of productivity and impact of research and scholarship. It is as if with the same (or decreasing) number of academics, there can be more publications, more students, more citations and more research income. Yet, since the number of tenure-track, permanent or even fixed-term positions is not increasing at the same rate as the number of PhD students, there is a scarcity of jobs, which in turn creates a hypercompetitive environment where researchers are competing to strive, or to simply survive. At the same time, scientific meritocracy promotes the idea that one's productivity and performance are one's achievements alone. Researchers are acculturated to recognise the symbolic and transactional values of metrics, which in turn put each individual as an “enterprising and competitive entrepreneur” (Esposito and Stark, 2019) and reproduce and reinforce “the entrepreneurial research culture” (Roumbanis, 2019) that upholds the “neoliberal notions of individualism and competitiveness” (Morrissey, 2013). Consequently, relationships are coded and marketed in transactional terms based on metrics, to the extent that “trust, integrity, care and solidarity are subordinated to regulation, control and competition” (Lynch, 2015, p. 195). The hypercompetitive market is intensified as metrics-based rankings are created and individuals and universities are compared and scrutinised continuously.

In essence, a competition has its winners and losers. The use of metrics fosters competition between individuals and institutions by benchmarking, rankings and creating hierarchies. Since these competitions are continuous, meaning one's place on a ranking can be surpassed by others there is the tendency to increase productivity and performance simply to avoid falling out of place. Not only that metrics become productivity and performance targets, but they are also the rule of the competitions. Notwithstanding these competitions have narrow definitions of “excellence” and “impact”, the pursuit of metrics seemingly cannot be resisted. Once the academic market becomes a ground for competition, then there is a danger that we are depreciating the value of humility and solidarity. As Halffman and Radder (2015) notes, the competition to publish in top ranked journals “constitutes the umpteenth step in the process of individualisation and the erosion of solidarity” (p. 174). Since competition, by its very nature, does not promote equality, the pursuit of metrics is not an antidote to systemic inequalities; rather, it perpetuates epistemic injustice in our recorded knowledge.

Epistemic injustice and information

In her book, Epistemic Injustice, Fricker (2007) examines how injustice can be done to a knower: “Testimonial injustice occurs when prejudice causes a hearer to give a deflated credibility to a speaker's word; hermeneutical injustice occurs at a prior stage, when a gap in collective interpretive resources puts someone at an unfair disadvantage when it comes to making sense of their social experience” (p. 1). Epistemic injustice is related to the perception of prejudicial stereotypes, for example, the ways by which the testimonials of ethnic minorities are often degraded, if not outright rejected, in social situations for who they are, not what they say. Put another way, their testimonials are not believed or recognised as a knower; what they say are often trivialized or simply do not matter. Fricker (2007) maintains that testimonial injustice is “systematically connected to other kinds of actual and potential injustice” (p. 27).

When metrics propagate competition and leverage power relations, they are antithetical to fair and just knowledge practices. That is to say, they can exclude knowledge claims of the marginalized and the suppressed. Metrics become a badge of honour because one's works deserve the “vote” in terms of citations, or one's productivity in terms of the number of publications in a hypercompetitive and meritocratic academic market. The presumed objectivity of metrics leads to the belief that scholarly works would be evaluated and judged solely based on their novelty, originality and contributions. In reality, the visibility and reception of scholarly works is probably not only a matter of its novelty or potential contributions. For one, the reasons that works of women and ethnic minorities are read less and cited less can be due to the perception of quality based on systemic and unconscious biases, or even outright discrimination. One may make the decision to read or cite an article judging by the name(s) of the author(s) and perhaps their affiliations, that is, rather than the contents of the articles. And if one looks up a person on Google Scholar, for example, women and ethnic minorities tend to have lower number of publications, citations and, as a result, a lower h-index. Hence, it would seem that their quality of work is “low” and their performance unsatisfactory, which further decreases the chance of these works to be taken seriously. In some cases, these works can be subject to manipulation such as “bounding” and “non-attribution” formulated by Bacevic (2021): when a speaker's knowledge claim is interpreted by hearers as constrained based on the speaker's perceived identity, social position or a combination of such characteristics and when a knowledge claim is not credited to the author. Similarly, those who are working on emerging, controversial or marginal topics can be subject to unjust assessments because their works are deemed to be of lesser quality and significance and are unworthy of being supported by grants and employment. The use of metrics leads to epistemic injustice because they cause the hearer—evaluators and potential readers—to give a deflated level of credibility of individuals and works that are ranked “lower”.

The academic hierarchies ordered by metrics are seemingly objective, implying that the competition is fair and just and that the contenders are competing on equal footing. It is an exemplar of meritocracy: if one's work is of good quality, ground breaking, then your metrics should be high and that's the proof for furthering one's fame, prestige and performance. The pursuit of metrics does not prompt or encourage one to invite or collaborate with those with weak indicators. As researchers are driven to increase productivity and maximise impact, there is no room for achieving equality and diversity in knowledge production. As every colleague becomes a competitor, researchers set aside considerations about the conditions that may disadvantage some. Therefore, an academic reward system that is based heavily on metrics is not fair and just, not only that metrics do not necessarily represent quality or impact, but they also downplay the values of collaboration and collegiality. When it is acceptable and understandable that a marathon runner does not stop to help a fellow runner when she is aiming to break the world record or personal best, the long, unceasing game of metrics, however, can empty us of compassion and empathy, especially in face of precarious contracts and redundancies. Compounded by the commercial interests that solidify the significance of productivity and impact, metrics promote and propagate competition and social acceleration. Researchers are prompted to be achievers and the system merits those who deserve them. It is as if success in the pursuit of metrics is only a matter of hard work, talent or even luck, notwithstanding systemic biases and inequalities have put some in a disadvantageous position to begin with, and the use of metrics as arbitrator of success can reproduce and reinforce existing power relationships and systemic inequalities in the society at large.

Further, these systemic biases and inequalities can be exacerbated by search algorithms as they use the number of citations (Google Scholar) or CiteScore (Scopus) to rank the results. Presumably, search results based on citation counts or scores aim to prioritise the most relevant results pertaining to a topic. In other words, the more a publication has been cited, the more it is deemed relevant. In effect, publications with the highest citations are likely to receive more citations, whereas publications that are less read and less cited in the first instance are not likely to attract attention or to claim relevance. Coupled with the perception of quality based on socio-cultural, political and religious factors, the voices of minorities can be buried as search algorithms prioritise citation counts. Metrics, while performing their functions as indicators of research quality and impact, can render some works more visible and others undiscoverable. On social media platforms, search algorithms are designed to keep users' attention on a platform for commercial purposes. Google Scholar or Scopus does not aim to sell ads, but nevertheless solidifies the legitimacy of and the desire for metrics as the ranked list of search results intensifies competition. Publications retrieved and ranked at the top are presumably the best works on a topic, while those with few citations are deemed irrelevant; despite the fact that citation counts and relevance are not the same and, that publications with no or fewer citations can be based on reasons other than quality and impact. In fact, studies have shown gender-based biases begin at the peer review process (Helmer et al., 2017; Squazzoni et al., 2021) and these biases are equally possible post-publication. When there is ample evidence about gender and racial disparities in research productivity and performance (e.g. Bertolero et al., 2020; Ginther et al., 2018; Larivière et al., 2013; Meho, 2021; Rachid et al., 2021; Sebo et al., 2021) and when factors such as caring responsibilities and work conditions are considered, the very use of metrics in the academic reward system must be taken into account. It is because metrics can be a steering medium that perpetuates systemic biases and injustices. Metrics serve different functions in the system of knowledge production and as such their meanings vary as a target in a strategic research plan, a ranking mechanism, a search algorithm or a criterion in a university ranking (Ma, 2021b). Nevertheless, they prompt competitions and they can render some works invisible and invaluable.

The scholarly world is not immune to epistemic injustice—some are given deflated credibility because of who they are or how they look, not what they say. Presumably, metrics should create a fair and just comparison regardless of one's backgrounds. However, that can be far from the truth as the naïve assumption of metrics' objectivity neglects the fact that when metrics become instrumental in rankings of various sorts, they enrol researchers and institutions in a fierce competition, an unending race, that is, until/unless one calls quit [5]. As the priority is to win, systemic biases are less likely to be shaken even though metrics are not designed to create injustices between rich and poor, Black and White, or men and women. Problems and issues pertaining to equality and diversity are often discussed and addressed outside of the academic reward system; however, the very use of metrics and the pressure of rankings are seldom considered regarding epistemic injustice: the devaluing or outright dismissal of knowledge claims by those whose metrics have been disadvantaged by their locations and backgrounds. As metrics leverage who can speak and who will be heard, epistemic injustice is reflected in recorded knowledge and what we consider to be information.

Conclusion: who is responsible for responsible metrics?

The first step is to measure whatever can easily be measured. This is OK as far as it goes. The second step is to disregard that which cannot be easily measured or to give it an arbitrary quantitative value. This is artificial and misleading. The third step is to presume that what cannot be measured easily really is not important. This is blindness. The fourth step is to say that what cannot be easily measured really does not exist. This is suicide. – The Macnamara Fallacy [6]

In his paper, Tobi (2020) proposes that “any practice that involves epistemic injustice is antithetical to the fair-minded pursuit of knowledge” and that “any practice that is antithetical to the fair-minded pursuit of knowledge should be reversed for epistemic reasons (p. 261). What can we do to reverse the metrics-driven research culture to ensure fair-minded pursuit of knowledge? That is, when metrics are instrumental in perpetuating competition by means of league tables and rankings and when a hypercompetitive research culture is negligent of epistemic injustice? Metrics are not good or bad in-and-themselves. However, the overdose and misuse of metrics in the system of knowledge production makes metrics a culprit because without metrics there can be no rankings and the problems of misconduct and malpractices, not to mention the distress and despair in research culture. However, it can be argued that metrics are set up as a scapegoat because metrics do not have agency and they do not actually do anything: universities and research institutions can depend less on data analytics and research intelligence, researchers can pay no or little attention to the number of views, downloads and citations, while there can be more awards and rewards for epistemic modesty (Carson, 2020); furthermore, there can be a more humble form of knowledge production that is open to marginalised and alternative perspectives. Metrics are not a cause of the vices in research culture, but we must consider the purposes and intentions of the scoreboards of metrics: are they there to provide an objective, neutral evaluation of universities and researchers? Or are they used to incite competition for an increased appetite for data and products of commercial enterprises? Or, are they used as a technocratic device in a metric society (see, for example, Davies, 2014; Mau, 2019)?

Whatever the purposes and intentions behind the use of metrics, it is worth noting that there cannot be an unbounded increase in scholarly outputs and impacts, just as impossible as an infinite economic growth. When the GDP (Gross Domestic Product) does not account for the damages to our environment, publication- and citation-based metrics do not account for bullying, harassment, discrimination or mental health and wellbeing of researchers. When the pursuit of metrics is ingrained in the research culture, competition becomes inevitable. And when metrics are embedded in search algorithms, not only does the rich get richer in terms of metrics, the algorithms also amplify the dominant, popular voices while effectively devaluing other knowledge claims.

Yet, bibliometric studies can debunk injustice and inequality in research and scholarship. Without which we will lack evidence to demonstrate the disparity in recognition and performance due to structural, systemic reasons. The dual roles of metrics as an assessment tool and as research data creates a conundrum: when metrics are used to decide who can stay and who must go when creating the (short) lists of candidates and redundancy [7], they are regarded as objective indicators that reflect research performance either in the sense of research quality or impact, or both. It is as if there can be no caveats or uncertainties that these metrics speak for an individual's research capacity and research performance; it is as if metrics represent the worth and value of researchers and their work. However, when metrics are used as research data—evidence—of gender gap and ethnic disparity in research productivity and performance, they are explicitly telling us that metrics are not necessarily representative of the quality or impact of these individuals' publications. It is because there can be other reasons that higher metrics are not gained and rewarded. These other reasons are not concerned with the contents and qualities, but rather, the cultural norms and social structures that affect why certain works are less read, less cited, devalued, or even stolen.

So, when considering “responsible metrics”, decision makers must attend to the structural discrimination and various forms of epistemic injustice (Fricker, 2007; Bacevic, 2021) in scholarly communication and research and publication practices. Metrics can be used as evidence insofar as they are not manipulated to the extent that they become unreliable: the gamification of metrics entails that metrics are not useable as research data for bibliometric studies, on the one hand, and as indicators of research quality and impact, on the other. As Wouters (2018) states, informetrics and scientometrics cannot be oblivious to the connections and the feedback processes that result from research assessments. Since bibliometric data such as citations can give voice to problems of power, diversity, inequality and injustice in research assessment and scholarly communication, it is important that metrics remain, at an aggregate level, useful and trustworthy data that can be analysed to reveal systemic and structural inequalities in academic research and scholarly communication. If metrics are to remain reliable data for bibliometric and scientometric studies, then research and publication practices cannot be steered by systemic influences imposed upon research and scholarship – that is, metrics cannot be used as a technocratic apparatus that stirs competitions by way of rankings and indicators to the extent that metrics are gamed and manipulated.

Notes

1. Publication- and citation-based metrics include the number of publications and citations as well as indicators such as journal impact factor (JIF), Citescore, Field Weighted Citation Impact FWCI), Source Normalized Impact per Paper (SNIP).

2. The Bibliomagician blog includes examples of statements of responsible metrics. Available at https://thebibliomagician.wordpress.com/statements-of-responsible-metrics-2

3. Similar arguments have been made in studies in higher education and research evaluation by many see, for example, Inquiring Into Academic Timescape, edited by Filip Vostal (2021).

4. The Times Higher Education, for example, has developed impact rankings, young university rankings as well we university rankings by geographical regions.

5. In fact, the academic “quit lit” is an emerging genre as noted in the LSE Impact Blog by Lara McKenzie - https://blogs.lse.ac.uk/impactofsocialsciences/2021/08/18/reading-academic-quit-lit-how-and-why-precarious-scholars-leave-academia/

6. The Macnamara Fallacy has been discussed in Charles Handy's The Empty Raincoat (2002) and Seamus O'Mahony's Can Medicine be Cured? The Corruption of a Profession (2019).

7. A recent high-profile case is the notification of redundancy of forty-seven academic staff in the University of Liverpool in January 2021. The criteria of redundancy were grant income targets and Scopus's Fields Weighted Citation Impact (FWCI). The University of Liverpool is a signatory of DORA and was/is expected to adhere to the principles of responsible metrics (Else, 2021; Whyte, 2021).