BibSonomy :: publication ::

tag user group author concept BibTeX key search:all search:diego_ma

The blue social bookmark and publication sharing system.

entry of diego_ma:

(0)

This publication has not been reviewed yet.

rating distribution

average user rating

The average rating is computed over all reviews. However, some of them may be invisible to you due to the visibility setting chosen by the reviewers.

(0.0 of 5.0 based on 0 reviews)

Answer Extraction -- Towards Better Evaluations of NLP Systems

by: Rolf Schwitter, Diego Mollá, Rachel Fournier, and Michael Hess

(2000) , p. 20-27.

Abstract

We argue that reading comprehension tests are not particularly suited for the evaluation of NLP systems. Reading comprehension tests are specifically designed to evaluate human reading skills, and these require vast amounts of world knowledge and common-sense reasoning capabilities. Experience has shown that this kind of full-fledged question answering QA over texts from a wide range of domains is so difficult for machines as to be far beyond the present state of the art of NLP. To advance the field we propose a much more modest evaluation set-up, viz. Answer Extraction AE over texts from highly restricted domains. AE aims at retrieving those sentences from documents that contain the explicit answer to a user query. AE is less ambitious than full-fledged QA but has a number of important advantages over QA. It relies mainly on linguistic knowledge and needs only a very limited amount of world knowledge and few inference rules. However, it requires the solution of a number of key linguistic problems. This makes AE a suitable task to advance NLP techniques in a measurable way. Finally, there is a real demand for working AE systems in technical domains. We outline how evaluation procedures for AE systems over real world domains might look like and discuss their feasibility.

BibTeX record

@inproceedings{Schwitter:2000,
  abstract = {We argue that reading comprehension tests are not particularly suited for the evaluation of NLP systems. Reading comprehension tests are specifically designed to evaluate human reading skills, and these require vast amounts of world knowledge and common-sense reasoning capabilities. Experience has shown that this kind of full-fledged question answering (QA) over texts from a wide range of domains is so difficult for machines as to be far beyond the present state of the art of NLP. To advance the field we propose a much more modest evaluation set-up, viz. Answer Extraction (AE) over texts from highly restricted domains. AE aims at retrieving those sentences from documents that contain the explicit answer to a user query. AE is less ambitious than full-fledged QA but has a number of important advantages over QA. It relies mainly on linguistic knowledge and needs only a very limited amount of world knowledge and few inference rules. However, it requires the solution of a number of key linguistic problems. This makes AE a suitable task to advance NLP techniques in a measurable way. Finally, there is a real demand for working AE systems in technical domains. We outline how evaluation procedures for AE systems over real world domains might look like and discuss their feasibility.},
  added-at = {2008-01-29T09:03:47.000+0100},
  author = {Schwitter, Rolf and Moll{\'a}, Diego and Fournier, Rachel and Hess, Michael},
  biburl = {http://www.bibsonomy.org/bibtex/2449b4548c23384e9a02234898bdc9715/diego_ma},
  crossref = {ZZZ-Brill:2000},
  interhash = {d1873a08ccf2b428f2a6090d1c54d27e},
  intrahash = {449b4548c23384e9a02234898bdc9715},
  keywords = {answer_extraction evaluation molla_publication},
  pages = {20-27},
  timestamp = {2008-01-29T09:03:47.000+0100},
  title = {Answer Extraction -- Towards Better Evaluations of {NLP} Systems},
  year = 2000
}

Endnote record

%0 Conference Paper
%1 Schwitter:2000
%A Schwitter, Rolf
%A Moll{\'a}, Diego
%A Fournier, Rachel
%A Hess, Michael
%D 2000
%K 
%P 20-27
%T Answer Extraction -- Towards Better Evaluations of {NLP} Systems
%X We argue that reading comprehension tests are not particularly suited for the evaluation of NLP systems. Reading comprehension tests are specifically designed to evaluate human reading skills, and these require vast amounts of world knowledge and common-sense reasoning capabilities. Experience has shown that this kind of full-fledged question answering (QA) over texts from a wide range of domains is so difficult for machines as to be far beyond the present state of the art of NLP. To advance the field we propose a much more modest evaluation set-up, viz. Answer Extraction (AE) over texts from highly restricted domains. AE aims at retrieving those sentences from documents that contain the explicit answer to a user query. AE is less ambitious than full-fledged QA but has a number of important advantages over QA. It relies mainly on linguistic knowledge and needs only a very limited amount of world knowledge and few inference rules. However, it requires the solution of a number of key linguistic problems. This makes AE a suitable task to advance NLP techniques in a measurable way. Finally, there is a real demand for working AE systems in technical domains. We outline how evaluation procedures for AE systems over real world domains might look like and discuss their feasibility.