by Arjen de Vries (CWI Amsterdam)
Information retrieval evaluation assesses how well systems identify information objects relevant to the user's information need. Traditional evaluations have used the following working definition of relevance: If you were writing a report on the subject of the topic and would use the information contained in the document in the report, then the document is relevant. Here, a document is judged relevant if any piece of it is relevant (regardless of how small that piece is in relation to the rest of the document).
Many realistic user tasks seem however better characterised by a different notion of relevance. Often, users search for specific entities instead of just any type of documents. Example information needs include "Countries where one can pay with the euro" or "Impressionist art musea in The Netherlands". To evaluate retrieval systems handling these typed information needs, the Initiative for Evaluation of XML Retrieval (INEX) has started to build a test collection for entity retrieval in Wikipedia, where the entities are assumed to correspond to Wikipedia entries. The talk discusses the consequences of modifying the definition of relevance on retrieval system evaluation.