An Overview of Evaluation Methods in TREC Ad Hoc Information Retrieval and TREC Question Answering
Evaluation of Text and Speech Systems, chapter 6, (2007)DOI: 10.1007/978-1-4020-5817-2_6
Abstract
This chapter gives an overview of the current evaluation strategies and problems in the fields of information retrieval (IR) and question answering (QA), as instantiated in the Text Retrieval Conference (TREC). Whereas IR has a long tradition as a task, QA is a relatively new task which had to quickly develop its evaluation metrics, based on experiences gained in IR. This chapter will contrast the two tasks, their difficulties, and their evaluation metrics. We will end this chapter by pointing out limitations of the current evaluation strategies and potential future developments.