Practical Methods for Evaluating Enterprise Search

by David Hawking, Funnelback Internet and Enterprise Search & Australian National University

Measurement of the quality of result pages returned by enterprise search facilities can be important for a number of reasons: quantifying how useful they are, identifying problems, tuning for performance, and making comparisons to guide purchasing decisions. Side-by-side, in-situ, A versus B comparisons have strong advantages but they are no use for routine tuning, such as a search engine company needs to carry out. The C-TEST formats and associated tools are intended to capture many of the advantages of side-by-side comparisons but in a form suitable for batch tuning. If time permits, the talk will outline some of the many reasons why particular searches fail and discuss evaluation methods for tuning and comparing essential enterprise search components other than the ranking algorithm.