Technology assisted research assessment: algorithmic bias and transparency issues

Abstract

Purpose

Technology is sometimes used to support assessments of academic research in the form of automatically generated bibliometrics for reviewers to consult during their evaluations or by replacing some or all human judgements. With artificial intelligence (AI), there is increasing scope to use technology to assist research assessment processes in new ways. Since transparency and fairness are widely considered important for research assessment and AI introduces new issues, this review investigates their implications.

Design/methodology/approach

This article reviews and briefly summarises transparency and fairness concerns in general terms and through the issues that they raise for various types of Technology Assisted Research Assessment (TARA).

Findings

Whilst TARA can have varying levels of problems with both transparency and bias, in most contexts it is unclear whether it worsens the transparency and bias problems that are inherent in peer review.

Originality/value

This is the first analysis that focuses on algorithmic bias and transparency issues for technology assisted research assessment.

Citation

Thelwall, M. and Kousha, K. (2023), "Technology assisted research assessment: algorithmic bias and transparency issues", Aslib Journal of Information Management, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/AJIM-04-2023-0119