Evaluating web search engines

Citation

Lewandowski, D. (2011), "Evaluating web search engines", Online Information Review, Vol. 35 No. 6. https://doi.org/10.1108/oir.2011.26435faa.001

Evaluating web search engines

Article Type: Guest editorial From: Online Information Review, Volume 35, Issue 6

Every month, more than 130 billion queries worldwide are entered into the search boxes of general-purpose web search engines (ComScore, 2010). This enormous number shows that web searching is not only a large business, but also that many people rely on the search engines’ results when researching information. Additionally users largely rely on Google, which enjoys a 65 percent market share in the USA (ComScore, 2011) and has a greater than 90 percent market share in many European countries (see Lunapark, 2011). However this focus on Google may obscure the fact that the total number of search queries executed on all major search engines is continuously growing, i.e. even search engines that fall behind in terms of market share grow in terms of the total number of queries.

Speaking of the importance of web search engines in general, it is surprising to find that this importance is quite contrary to the users’ approach towards them. In general users are competent neither in the use of search engines nor in evaluating their results (Machill et al., 2004).

An obvious goal of all search engine evaluation efforts is to generate better systems. This goal is of major importance to the search engine vendors who can directly apply evaluation results to develop better ranking algorithms. However there are a multitude of other goals in search engine evaluation that are often forgotten. This failure has to do with the assumption that the developers of search systems themselves need to agree to have their systems evaluated and to let them be compared in a lab setting, such as in the TREC evaluation initiative. However one can see that none of the major search engine vendors takes part in these evaluations. Moreover it is questionable whether the results from these relatively small-scale evaluation efforts also hold true when scaled to web-size.

Comparing web search engines may answer an important question, i.e. do users choose to use a search engine (e.g. Google) solely because of its higher quality, or are other reasons responsible for this choice? When millions of Euros are invested in developing search technology in government-funded initiatives such as Theseus in Germany and Quaero in France, it would be interesting to know whether the proposed higher quality of such efforts would be useful in terms of users’ choices or whether a simple shift in market shares between the existing major search engines would be enough to give the users a good choice between high-quality search engines.

Another question one could ask is why the quality of search engines, and especially the quality of their results, is of such importance. While it is evident that users would like to use a search engine that produces relevant results, the usefulness of direct comparisons might not be that evident at first, or as Péter Jacsó put it in his column in this journal, “In the ideal world one perfect search engine would suffice” (Jacsó, 2008, p. 864). However this statement overlooks the possibility that a choice among search engines would be desirable for the users, as search engines rank documents differently and, therefore, return different results for the same query (Spink et al., 2006). Accordingly users might see the results from an additional search engine as a “second opinion” on their query.

Regarding “search neutrality”, a topic that has generated greater interest recently, it is at least questionable that the major web search engines always show the “best” results for a query at the top of the list they return. Some researchers have found that the search engines favour their own offerings within their results lists (Höchstötter and Lewandowski, 2009; Edelman, 2010; Edelman and Lockwood, 2011). Thus some monitoring of the search engines would be desirable from a societal point of view. Moreover it is not desirable that just one search engine should determine what users get to see.

In this special issue we have methodological as well as empirical papers. However the major differences lie in their approaches to evaluating search engines in meeting “general-purpose” or specialised information needs. As web search engines are used for so many purposes, we need both types of studies. It is greatly important that we know how good search engines are in general, but we also need to know more about their ability to answer specialised queries.

Although evaluating the results of web search engines is of great importance, one should bear in mind that, regarding the results, quality is just one factor in overall search engine quality measurements. According to Lewandowski and Höchstötter (2008) search engine quality measurement can be grouped into four major areas:

1. 1.

   index quality;

2. 2.

   quality of the results;

3. 3.

   quality of the search features; and

4. 4.

   search engine usability.

Finally we want to stress that the results from the studies of web search engines can, at least to some extent, be adapted to other information systems. We need to develop evaluation methods for the quality of the results of web site searches, of intranet searches, and of many other search purposes. The results from the studies presented in this special issue can help show that a variety of possibilities exists and that there is not only one general approach to evaluating search results.

Acknowledgements

The guest editor would like to thank all the reviewers for their good work and timely responses.

Dirk Lewandowski Professor in the Department of Information, Hamburg University of Applied Sciences, Hamburg, Germany.