INEX 2005 Alternative Evaluation

There is a strong compatibility argument for researchers to use the same methods as each other unless there is very good reason to depart from the norm.

Stephen Robertson, "Evaluation in information retrieval". European Summer School on Information Retrieval (ESSIR), Lecture Notes in Computer Science 1980, Springer, pages 81-92, 2000. 

Icon  Name                    Last modified      Size  Description
[DIR] Parent Directory                             -   
[TXT] README.html             15-Nov-2005 09:34  1.3K  
[DIR] cityuni/                10-Jul-2006 14:41    -   
[DIR] cmu/                    10-Jul-2006 14:41    -   
[DIR] ibmhaifa/               10-Jul-2006 14:42    -   
[DIR] irit/                   10-Jul-2006 14:42    -   
[DIR] lip6/                   10-Jul-2006 14:42    -   
[DIR] maxplanck/              10-Jul-2006 14:42    -   
[DIR] ouc/                    10-Jul-2006 14:42    -   
[DIR] qutau/                  10-Jul-2006 14:43    -   
[DIR] rmit/                   10-Jul-2006 14:43    -   
[DIR] uamsterdam/             10-Jul-2006 14:43    -   
[DIR] uberkeley/              10-Jul-2006 14:44    -   
[DIR] ucalif/                 10-Jul-2006 14:44    -   
[DIR] uhebrew/                10-Jul-2006 14:44    -   
[DIR] uhelsinki/              10-Jul-2006 14:44    -   
[DIR] ukyung/                 10-Jul-2006 14:45    -   
[DIR] uminnesota/             10-Jul-2006 14:45    -   
[DIR] umontes/                10-Jul-2006 14:45    -   
[DIR] uniKaislau/             10-Jul-2006 14:46    -   
[DIR] uniisys/                10-Jul-2006 14:45    -   
[DIR] utampere/               10-Jul-2006 14:46    -   
[DIR] utrecht/                10-Jul-2006 14:46    -   
[DIR] utwente/                10-Jul-2006 14:46    -   
[DIR] uwaterloo/              10-Jul-2006 14:46    -   
[DIR] valoria/                10-Jul-2006 14:46    -   
[DIR] xtra-utrecht/           10-Jul-2006 14:46    -   
[DIR] xtra-utwente/           10-Jul-2006 14:47    -   
[DIR] xtra-utwente2004-2/     10-Jul-2006 14:47    -   
[DIR] xtra-utwente2004/       10-Jul-2006 14:47    -   

This directory contains alternative evaluation results for the INEX 2005
adhoc runs. Many thanks to the INEX organisation for making runs and 
assessments available at an early stage.

The files contain precision and overlap at several document cut-off values.
The files [x].full.txt contain measurements for every single topic including
a summary of the results (preceeded with '|') reporting the average precision
and average overlap (averaged over the topics). The files [x].summary.txt
only contain the precision and overlap averages. The files report precision
for 5 different quantisation functions. All quantisations are binary (either
relevant or not). The quantisation functions are: 0) very strict (which 
corresponds to the INEX 2005 official strict quantisiation), 1) strict, 2)
liberal, 3) exhaustiveness oriented, and 4) specificity oriented. For
more information, read the publication below.

Djoerd Hiemstra and Vojkan Mihajlovic, ``The Simplest Evaluation Measures 
for XML Information Retrieval that Could Possibly Work'', In Proceedings of 
the INEX 2005 Workshop on Element Retrieval Methodology, University of Otago,
New Zealand, July 30, ISBN 0-473-10071-1, pages 6-13, 2005
http://www.cs.utwente.nl/~hiemstra/papers/inexmw.pdf