The Combination and Evaluation of Query Performance Prediction Methods

by Claudia Hauff, Leif Azzopardi, and Djoerd Hiemstra

In this paper, we examine a number of newly applied methods for combining pre-retrieval query performance predictors in order to obtain a better prediction of the query’s performance. However, in order to adequately and appropriately compare such techniques, we critically examine the current evaluation methodology and show how using linear correlation coefficients (i) do not provide an intuitive measure indicative of a method’s quality, (ii) can provide a misleading indication of performance, and (iii) overstate the performance of combined methods. To address this, we extend the current evaluation methodology to include cross validation, report a more intuitive and descriptive statistic, and apply statistical testing to determine significant differences. During the course of a comprehensive empirical study over several TREC collections, we evaluate nineteen pre-retrieval predictors and three combination methods.

The paper will be presented at the 31st European Conference on Information Retrieval (ECIR), April 6-9, 2009 in Toulouse, France.

[download pdf]

2 Responses to “The Combination and Evaluation of Query Performance Prediction Methods”

  1. Guillaume Corriveau Says:
    The pdf file is not the paper describe by this abstract… I would like if you can send me the paper :The Combination and Evaluation of Query Performance Prediction Methods. Thanks Guillaume Corriveau
  2. Djoerd Hiemstra Says:
    Woops, thanks for noting. I corrected the link. Best, Djoerd.