A decision theoretic approach to combining information filtering

The purpose of this paper is to present an extension to a framework based on the information structure (IS) model for combining information filtering (IF) results. The main goal of the framework is to combine the results of the different IF systems so as to maximise the expected payoff (EP) to the user. In this paper we compare three different approaches to tuning the relevance thresholds of individual IF systems that are being combined in order to maximise the EP to the user. In the first approach we set the same threshold for each of the IF systems. In the second approach the threshold of each IF system is tuned independently to maximise its own EP (“local optimisation”). In the third approach the thresholds of the IF systems are jointly tuned to maximise the EP of the combined system (“global optimisation”).

An empirical evaluation is conducted to examine the performance of each approach using two IF systems based on somewhat different filtering algorithms (TFIDF, OKAPI). Experiments are run using the TREC3, TREC6, and TREC7 test collections.

The experiments reveal that, as expected, the third approach always outperforms the first and the second, and that for some user profiles, the difference is significant. However, operational goals argue against global optimisation, and the costs of meeting these operational goals are discussed.

One limitation is the assumption of independence of the IF systems: in real life systems usually use similar algorithms, so dependency might occur. The approach also tends to be examined with the assumption of dependency between systems.

The main practical implications of this study lie in the empirical proof that combination of filtering systems improves filtering results and the finding about the optimal combination methods for the different user profiles. Many filtering applications exist (e.g. spam filters, news personalisation systems, etc.) that can benefit from these findings.

The study presents and compares the contribution of three different combination methods of filtering systems to the improvement of filtering results It empirically shows the benefits of each method and draws important conclusions about the combination of filtering systems.