BibSonomy publications for /user/diego_ma/evaluationWed Oct 21 23:50:43 CEST 2009Proc BioNLP 2009171-178Evaluation of the Clinical Question Answering Presentation.2009biomedical question_answering evaluation Question answering is different from information retrieval in that it attempts to answer questions by providing summaries from numerous retrieved documents rather than by simply providing a list of documents that requires users to do additional work. However, the quality of answers that question answering provides has not been investigated extensively, and the practical approach to presenting question answers still needs more study. In addition to factoid answering using phrases or entities, most question answering systems use a sentence- based approach for generating answers. However, many sentences are often only meaningful or understandable in their context, and a passage-based presentation can often provide richer, more coherent context. However, passage-based presentations may introduce additional noise that places greater burden on users. In this study, we performed a quantitative evaluation on the two kinds of presentation produced by our online clinical question answering system, AskHERMES (http://www.AskHERMES.org). The overall finding is that, although irrelevant context can hurt the quality of an answer, the passage-based approach is generally more effective in that it provides richer context and matching across sentences.Fri Aug 14 09:50:03 CEST 2009Proc. Pacific Symposium on Biocomputing328-339A Cognitive Evaluation of Four Online Search Engines for Answering Definitional Questions Posed By Physicians2007search inf-retr biomedical evaluation The Internet is having a profound impact on physicians' medical decision making. One recent survey of 277 physicians showed that 72\% of physicians regularly used the Internet to research medical information and 51\% admitted that information from web sites influenced their clinical decisions. This paper describes the first cognitive evaluation of four state-of-the-art Internet search engines: Google (i.e., Google and Scholar.Google), MedQA, Onelook, and PubMed for answering definitional questions (i.e., questions with the format of "What is X?") posed by physicians. Onelook is a portal for online definitions, and MedQA is a question answering system that automatically generates short texts to answer specific biomedical questions. Our evaluation criteria include quality of answer, ease of use, time spent, and number of actions taken. Our results show that MedQA out- performs Onelook and PubMed in most of the criteria, and that MedQA surpasses Google in time spent and number of actions, two important efficiency criteria. Our results show that Google is the best system for quality of answer and ease of use. We conclude that Google is an effective search engine for medical definitions, and that MedQA exceeds the other search engines in that it provides users direct answers to their questions; while the users of the other search engines have to visit several sites before finding all of the pertinent information.Fri May 15 06:50:52 CEST 2009Recent Advances in Natural Language Processing for Biomedical Applications Special IssueInternational Journal of Medical InformaticsJune6430--442Evaluation of two dependency parsers on biomedical corpus targeted at protein-protein interactions752006parsers biomedicine evaluation We present an evaluation of Link Grammar and Connexor Machinese Syntax, two major broad-coverage dependency parsers, on a custom hand-annotated corpus consisting of sentences regarding protein-protein interactions. In the evaluation, we apply the notion of an interaction subgraph, which is the subgraph of a dependency graph expressing a protein-protein interaction. We measure the performance of the parsers for recovery of individual dependencies, fully correct parses, and interaction subgraphs. For Link Grammar, an open system that can be inspected in detail, we further perform a comprehensive failure analysis, report specific causes of error, and suggest potential modifications to the grammar. We find that both parsers perform worse on biomedical English than previously reported on general English. While Connexor Machinese Syntax significantly outperforms Link Grammar, the failure analysis suggests specific ways in which the latter could be modified for better performance in the domain.Fri Jan 16 00:48:33 CET 2009Proceedings of the GEAF 2007 Workshop21 pagesCSLI Studies in Computational Linguistics OnlineTowards Framework-Independent Evaluation of Deep Linguistic Parsers2007parsers evaluation This paper describes practical issues in the framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependency-based syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)'s Grammatical Relations and de Marneffe et al. (2006)'s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation.Wed Mar 12 04:43:46 CET 2008Proc. ACL workshop on Text Summarization Branches Out10ROUGE: A Package for Automatic Evaluation of summaries2004summarisation evaluation COMP448 ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. It includes measures to automatically determine the quality of a summary by comparing it to other (ideal) summaries created by humans. The measures count the number of overlapping units such as n-gram, word sequences, and word pairs between the computer-generated summary to be evaluated and the ideal summaries created by humans. This paper introduces four different ROUGE measures: ROUGE-N, ROUGE-L, ROUGE-W, and ROUGE-S included in the ROUGE summarization evaluation package and their evaluations. Three of them have been used in the Document Understanding Conference (DUC) 2004, a large-scale sum- marization evaluation sponsored by NIST.Wed Mar 12 04:42:41 CET 2008Proc. HLT-NAACL8 pagesAutomatic Evaluation of Summaries Using N-Gram Co-occurrence Statistics2003summarisation evaluation COMP448 Following the recent adoption by the machine translation community of automatic evaluation using the BLEU/NIST scoring process, we conduct an in-depth study of a similar idea for evaluating summaries. The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.Tue Jan 29 09:03:47 CET 2008ZZZ-Brill:200020-27Answer Extraction -- Towards Better Evaluations of {NLP} Systems2000answer_extraction evaluation molla_publication We argue that reading comprehension tests are not particularly suited for the evaluation of NLP systems. Reading comprehension tests are specifically designed to evaluate human reading skills, and these require vast amounts of world knowledge and common-sense reasoning capabilities. Experience has shown that this kind of full-fledged question answering (QA) over texts from a wide range of domains is so difficult for machines as to be far beyond the present state of the art of NLP. To advance the field we propose a much more modest evaluation set-up, viz. Answer Extraction (AE) over texts from highly restricted domains. AE aims at retrieving those sentences from documents that contain the explicit answer to a user query. AE is less ambitious than full-fledged QA but has a number of important advantages over QA. It relies mainly on linguistic knowledge and needs only a very limited amount of world knowledge and few inference rules. However, it requires the solution of a number of key linguistic problems. This makes AE a suitable task to advance NLP techniques in a measurable way. Finally, there is a real demand for working AE systems in technical domains. We outline how evaluation procedures for AE systems over real world domains might look like and discuss their feasibility.Tue Jan 29 08:19:28 CET 2008BudapestProc. European Association for Computational Linguistics (EACL), workshop on Evaluation Initiatives in Natural Language ProcessingApril43-50Intrinsic versus Extrinsic Evaluations of Parsing Systems2003parsers evaluation AnswerFinder gram_rels molla_publication A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of \newcite{Carroll:1998} for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant.Tue Jan 29 08:16:55 CET 2008In preparationIn Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction2002AnswerFinder parsers evaluation gram_rels molla_publication A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (\emph{in vitro} evaluations), and only in a few cases has the effect of different parsers in real applications been measured (\emph{in vivo} evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared \emph{in vitro} by converting the dependencies into grammatical relations and using the methodology of \newcite{Carroll:1998} for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared \emph{in vivo} within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely \emph{in vitro} evaluations.Fri Dec 14 02:49:13 CET 2007Seattle, WAProc. {ANLP/NAACL} 2000 Workshop on Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding SystemsProc. {ANLP/NAACL} 2000 Workshop on Reading Comprehension Tests as Evaluation for Computer-Based Language Understanding Systems2000evaluation answer_extraction question_answering Fri Dec 14 02:49:12 CET 2007Seattle, WAProc. {ANLP/NAACL} 2000 Workshop on Syntactic and Semantic Complexity in Natural Language Processing SystemsProc. {ANLP/NAACL} 2000 Workshop on Syntactic and Semantic Complexity in Natural Language Processing Systems2000evaluation Fri Dec 14 02:48:25 CET 2007Seattle, WA\myurl{http://www.gte.com/AboutGTE/gto/anlp-naacl2000/comprehension.html}Workshop on Reading Comprehension Texts as Evaluation for Computer-Based Language Understanding Systems2000evaluation Fri Dec 14 02:48:05 CET 2007Loughborough UniversityProc. Third Annual Computer Assisted Assessment Conference207-219Approaches to the Computerized Assessment of Free Text Responses1999evaluation question_answering The automated assessment of student's essays is regarded by many as the Holy Grail of computer aided assessment. Whilst a few people search for the grail, many more deny its existence. This paper describes the various approaches that have been taken over the last 40 years in an attempt to solve the problems involved with the computerized assessment of free text. The earliest approaches were founded in simple style analysis. Systems, such as Project Essay Grade (PEG), were developed upon the idea that certain surface features of an essay could be manipulated in such a way as to predict the grade that a human examiner would assign to an essay. Other methods, such as Latent Semantic Analysis (LSA), also take a statistical approach to marking, but focus on actual textual content, analyzing groupings and context. The Educational Testing Service (ETS) originally attempted to tackle the problem from a classification point of view whilst more recent work by ETS bears similarities to PEG in its statistical approach. Most of the methods currently being developed have been shown to be capable of generating essay scores that correlate with a human grader's scores at least as well as two human graders correlate with each other. A novel approach being adopted by the authors to allow the comparison of students' essays against a model answer involves the use of theories developed for inter-lingual machine translation. A few different methods for knowledge representation, and their current uses in machine translation are presented. Panlingua, an idea for knowledge representation developed by Chaumont Devin, is based on semantic network research. Using a system of nodes over four layers it attempts to model how the brain might translate from sensory patterns it sees or hears at the top level, through syntactic and semantic levels, to a representation of understanding at the deepest level. Another approach to machine translation involves work done by Bonnie Dorr at the University of Maryland based on Lexical Conceptual Structure (LCS) theory. This allows the knowledge represented in a text to be translated into a language independent data structure. These theories provide a way of representing knowledge that is not reliant on the surface syntax of the text representing the knowledge. This will hopefully allow 'fuzzy matching' of sentences which have different syntactic structures but similar semantic meaning. The authors will be looking at the possibility of grading essays via a comparison of these data structures.Fri Dec 14 02:47:27 CET 2007ERB-1051Extraction of Keyphrases from Text: Evaluation of Four Algorithms1997inf_retrieval evaluation This report presents an empirical evaluation of four algorithms for automatically extracting keywords keyphrases from documents. The four algorithms are compared using five different collections of documents. For each document, we have a target set of keyphrases, which were generated by hand. The target keyphrases were generated for human readers; they were not tailored for any of the four keyphrase extraction algorithms. Each of the algorithms was evaluated by the degree to which the...Fri Dec 14 02:46:13 CET 2007MontrealProc. Thirteenth Canadian Conference on Artificial IntelligenceThe Power of the {TSNLP}: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser2000parsers evaluation We show a diagnostic evaluation of DIPETT, a broad-coverage parser of English sentences. We consider the TSNLP suite as a diagnostic tool, and propose an alternative broader-coverage test suite of test sentences extracted from Quirk et al. We compare the diagnostic effectiveness of the two suites, and draw a few general conclusions. The evaluation results were used to make significant improvements to DIPETT.Fri Dec 14 02:46:12 CET 2007An Evaluation of a Rule-Based Parser of English Sentences2000parsers evaluation ... The thesis argues that a test suite for a broad coverage natural language parser must necessarily be systematic, broad in its coverage of phenomena tested, and corpus-like in its coverage of phenomenon interaction. A test suite of example sentences extracted from Quirk et al.'s comprehensive English grammar is proposed, and the results of evaluating DIPETT on that suite are compared with the evaluation results on a publicly available test suite, TSNLP (Test Suites for Natural Language Processing).Fri Dec 14 02:42:33 CET 2007Proc. SIGIR'05Evaluation of Resources for Question Answering Evaluation2005question_answering evaluation resources In contrast to traditional information retrieval systems, which return ranked lists of documents that users must manually browse through, a question answering system attempts to directly answer natural language questions posed by the user. Although such systems possess language processing capabilities, they still rely on traditional document retrieval techniques to generate an initial candidate set of documents. In this paper, we argue that document retrieval for question answering represents a different task than retrieving documents in response to more general retrospective information needs. Thus, to guide future system development, specialized question answering test collections must be constructed. We have shown that the current evaluation resources have major shortcomings, and to remedy the situation, we have manually created a small, reusable question answering test collection for research purposes. This article describes our methodology for building this test collection and discusses issues we encountered along the way regarding the notion of ?answer correctness?.Fri Dec 14 02:41:36 CET 2007Communications of the ACM173-79Evaluating Natural Language Processing Systems391996evaluation NLP Designing customized methods for testing various NLP systems may be costly and complex, but the resulting data is invaluable. Researchers now wonder if there are ways to share evaluation methodologies.Fri Dec 14 02:40:57 CET 2007ZZZ-LREC:2002Cooperation between Black Box and Glass box Approaches for the Evaluation of a Question Answering System577-584evaluation question_answering Fri Dec 14 02:39:10 CET 2007TR-291Evaluating Natural Language Processing Systems1993evaluation NLP This report presents a detailed analysis and review of NLP evaluation, in principle and in Practice. Part 1 examines evaluation concepts and establishes a framework for NLP system evaluation. This makes use of experience in the related area of information retrieval and the analysis also refers to evaluation in speech processing. Part 2 surveys significant evaluation work done so far, for instance in machine translation, and discusses the particular problems of generic system evaluation. The conclusion is that evaluation strategies and techniques for NLP need much more development, in particular to take proper account of the influence of system tasks and settings. Part 3 develops a general approach to NLP evaluation, aimed at methodologically-sound strategies for test and evaluation motivated by comprehensive performance factor identification. The analysis throughout the report is supported by extensive illustrative examples.