TREC: Experiment and Evaluation in Information Retrieval

Citation

Chowdhury, G. (2007), "TREC: Experiment and Evaluation in Information Retrieval", Online Information Review, Vol. 31 No. 5, pp. 717-718. https://doi.org/10.1108/14684520710832478

Systematic evaluation of text retrieval systems began with the Cranfield tests in the 1960s. Since then several information retrieval (IR) evaluation experiments have taken place throughout the world, and during this period many new IR techniques and models have been developed. However, these IR models and techniques were developed and tested with very small test collections compared to the large operational IR systems, and scalability of the new models and techniques and their effectiveness were questionable.

TREC, the text retrieval conference series, was conceived in the early 1990 s to face these challenges. The main objectives of TREC were to facilitate text retrieval research by creating large test collections and making them available to the interested IR researchers, and also to create a global research environment where IR researchers could develop novel IR experiments, retrieval measures and methodologies, and could share and build on the experience of each other. The first TREC conference took place in 1992, and since then the event has taken place every year. This book provides one of the most comprehensive accounts of TREC studies up to the 2003 TREC conference. Details of TREC experiments and conference papers are available from the TREC web site (http://trec.nist.gov/), and a number of publications reporting on TREC and the corresponding experiments and evaluation studies have appeared in the literature (see, for example, Annual Review of Information Science and Technology, Vol. 40, 2005, and various issues of journals like Information Processing and Management, Vol. 31 No. 3, 1995; Vol. 36 No. 1, 2000; and Information Retrieval, Vol. 5, 2002; etc.).

However, this book provides the most comprehensive and value‐added reports on TREC studies. The book comprises 17 chapters divided into three parts. Each chapter has been written by recognised experts who have been actively associated with TREC for several years. There are three chapters in Part 1. The first chapter provides an introduction to TREC, including its history and developments, various research tracks in TREC and so on. Chapter 2 discusses the nature and characteristics of the TREC collections, while Chapter 3 provides details of the various retrieval measures and approaches developed in course of the 12‐year period of TREC experiments.

The seven chapters in Part 2 report on the selected tracks of TREC, describing some specific IR experiments and evaluation techniques. Part 3 of the book contains seven chapters contributed by selected sets of research groups who have participated in the TREC series of experiments. Four chapters have been written by four active research groups discussing their research, such as the CIIR (Centre for Intelligent IR) at the University of Massachusetts, the Okapi research group at City University in London (currently at Microsoft Research), the SMART research group at Cornell University (currently at Sabir research), and the Probabilistic IR research group at the State University of New York. The three other chapters in Part 3 are from three research groups who used TREC to evaluate their systems and validate their approaches to IR. In the epilogue Karen Sparck Jones reflects on the TREC experiments and addresses some fundamental questions, such as how the IR experiments should be modified in the light of the experience gained through TREC to meet the IR challenges in the web environment.

Overall, this is one of the best books on IR, and most certainly so on retrieval evaluation, reporting on the developments in IR research and evaluation that have taken place throughout the world. Hence it will be an extremely valuable source of information for the present and future generations of IR researchers.