User evaluation of a task for shortlisting papers from researcher’s reading list for citing in manuscripts

Although many interventional approaches have been proposed to address the apparent gap between novices and experts for literature review (LR) search tasks, there have been very few approaches proposed for manuscript preparation (MP) related tasks. The purpose of this paper is to describe a task and an incumbent technique for shortlisting important and unique papers from the reading list (RL) of researchers, meant for citation in a manuscript.

A user evaluation study was conducted on the prototype system which was built for supporting the shortlisting papers (SP) task along with two other LR search tasks. A total of 119 researchers who had experience in authoring research papers participated in this study. An online questionnaire was provided to the participants for evaluating the task. Both quantitative and qualitative analyses were performed on the collected evaluation data.

Graduate research students prefer this task more than research and academic staff. The evaluation measures relevance, usefulness and certainty were identified as predictors for the output quality measure “good list”. The shortlisting feature and information cues were the preferred aspects while limited data set and rote steps in the study were ascertained as critical aspects from the qualitative feedback of the participants.

Findings point out that researchers are clearly interested in this novel task of SP from the final RL prepared during LR. This has implications for digital library, academic databases and reference management software where this task can be included to benefit researchers at the manuscript preparatory stage of the research lifecycle.