Understanding credibility judgements for web search snippets

It is well known that information behaviour can be biased in countless ways and that users of web search engines have difficulty in assessing the credibility of results. Yet, little is known about how search engine result page (SERP) listings are used to judge credibility and in which if any way such judgements are biased. The paper aims to discuss these issues.

Two studies are presented. The first collects data by means of a controlled, web-based user study (N=105). Studying judgements for three controversial topics, the paper examines the extent to which users agree on credibility, the extent to which judgements relate to those applied by objective assessors and to what extent judgements can be predicted by the users’ position on and prior knowledge of the topic. A second, qualitative study (N=9) utilises the same setup; however, transcribed think-aloud protocols provide an understanding of the cues participants use to estimate credibility.

The first study reveals that users are very uncertain when assessing credibility and their impressions often diverge from objective judges who have fact checked the sources. Little evidence is found indicating that judgements are biased by prior beliefs or knowledge, but differences are observed in the accuracy of judgements across topics. Qualitatively analysing think-aloud transcripts from participants think-aloud reveals ten categories of cues, which participants used to determine the credibility of results. Despite short listings, participants utilised diverse cues for the same listings. Even when the same cues were identified and utilised, different participants often interpreted these differently. Example transcripts show how participants reach varying conclusions, illustrate common mistakes made and highlight problems with existing SERP listings.

This study offers a novel perspective on how the credibility of SERP listings is interpreted when assessing search results. Especially striking is how the same short snippets provide diverse informational cues and how these cues can be interpreted differently depending on the user and his or her background. This finding is significant in terms of how search engine results should be presented and opens up the new challenge of discovering technological solutions, which allow users to better judge the credibility of information sources on the web.