Effectiveness of Google keyword suggestion on users’ relevance judgment: A mixed method approach to query expansion

One of the most effective ways information retrieval (IR) systems including Web search engines can improve relevance performance is to provide their users with tools for facilitating query expansion. Search engines such as Google provide users with keyword suggest tools. This paper aims to investigate users’ criteria in relevance judgment regarding Google’s keywords suggest tool and to see how such keywords would lead to more relevant results from the viewpoint of users.

Through a mixed method approach, quantitative and qualitative data were collected from 60 postgraduate students at Ferdowsi University of Mashhad, Iran, using four different instruments (questionnaire, thinking aloud technique, query logs and interviews).

Among other criteria, the “relation between suggested keywords and the information need” (with the mean rate of 3.53 of four) was considered the most important by searchers in selecting suggested keywords for query expansion. Also, the “relation between suggested Keywords and the retrieved items” (with the mean rate of 3.62) was considered the second most important criterion in judging the relevance of the retrieved results. The participants agreed that the suggested keywords by Google improved the retrieval relevance. The content analysis of the participants’ aloud-thinking sessions and the interviews approved such findings.

This research makes a contribution to the need of designers of IR systems regarding the use of add words for query expansion. It also helps librarians how to instruct searchers with expanding their queries to retrieve more relevant results. Another contribution of the study is the identification of a number of new relevance judgment criteria for Web-based environments.