A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant.

@inproceedings{Moll'a:2003b, author = {Mollá, Diego and Hutchinson, Ben}, title = {Intrinsic versus Extrinsic Evaluations of Parsing Systems}, booktitle = {Proc. European Association for Computational Linguistics (EACL), workshop on Evaluation Initiatives in Natural Language Processing}, publisher = {ACL}, address = {Budapest}, year = {2003}, pages = {43-50}, keywords = {parsers evaluation AnswerFinder gram_rels molla_publication}, abstract = {A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant.} }

%0 = inproceedings %A = Mollá, Diego and Hutchinson, Ben %B = Proc. European Association for Computational Linguistics (EACL), workshop on Evaluation Initiatives in Natural Language Processing %C = Budapest %D = 2003 %I = ACL %T = Intrinsic versus Extrinsic Evaluations of Parsing Systems

A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (in vitro evaluations), and only in a few cases has the effect of different parsers in real applications been measured (in vivo evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared in vitro by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared in vivo within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely in vitro evaluations.

@unpublished{Molla:anlp02:preparation, author = {Mollá, Diego and Hutchinson, Ben}, title = {In Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction}, year = {2002}, note = {In preparation}, keywords = {AnswerFinder parsers evaluation gram_rels molla_publication}, abstract = {A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (in vitro evaluations), and only in a few cases has the effect of different parsers in real applications been measured (in vivo evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared in vitro by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared in vivo within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely in vitro evaluations.} }

%0 = unpublished %A = Mollá, Diego and Hutchinson, Ben %D = 2002 %T = In Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction

@misc{Briscoe:2000, author = {Briscoe, Ted and Carroll, John}, title = {Grammatical Relation annotation}, year = {2000}, note = {http://www.cogs.susx.ac.uk/lab/nlp/ carroll/grdescription/index.html}, url = {http://www.cogs.susx.ac.uk/lab/nlp/carroll/grdescription/index.html}, keywords = {parsers gram_rels} }

%0 = misc %A = Briscoe, Ted and Carroll, John %D = 2000 %T = Grammatical Relation annotation %U = http://www.cogs.susx.ac.uk/lab/nlp/carroll/grdescription/index.html

In this paper we discuss cascaded Memory-Based grammatical relations assignment. In the first stages of the cascade, we find chunks of several types (NP,VP,ADJP,ADVP,PP) and label them with their adverbial function (e.g. local, temporal). In the last stage, we assign grammatical relations to pairs of chunks. We studied the effect of adding several levels to this cascaded classifier and we found that even the less performing chunkers enhanced the performance of the relation finder.

@inproceedings{Buchholz:1999, author = {Buchholz, Walter Daelemans Sabine}, title = {Cascaded Grammatical Relation Assignment}, booktitle = {Proceedings of EMNLP/VLC-99}, year = {1999}, pages = {239-246}, url = {http://ilk.kub.nl/~sabine/}, keywords = {gram_rels}, abstract = {In this paper we discuss cascaded Memory-Based grammatical relations assignment. In the first stages of the cascade, we find chunks of several types (NP,VP,ADJP,ADVP,PP) and label them with their adverbial function (e.g. local, temporal). In the last stage, we assign grammatical relations to pairs of chunks. We studied the effect of adding several levels to this cascaded classifier and we found that even the less performing chunkers enhanced the performance of the relation finder.} }

%0 = inproceedings %A = Buchholz, Walter Daelemans Sabine %B = Proceedings of EMNLP/VLC-99 %D = 1999 %T = Cascaded Grammatical Relation Assignment %U = http://ilk.kub.nl/~sabine/

We present a critical overview of the state-of-the-art in parser evaluation methodologies and metrics. A discussion of their relative strengths and weaknesses motivates a new -- and we claim more informative and generally applicable -- technique of measuring parser accuracy, based on the use of grammatical relations. We conclude with some preliminary results of experiments in which we use this new scheme to evaluate a robust parser of English.

@inproceedings{Carroll:1998, author = {Carroll, John and Briscoe, Ted and Sanfilippo, Antonio}, title = {Parser Evaluation: a Survey and a New Proposal}, booktitle = {Proc. LREC98}, year = {1998}, url = {http://citeseer.nj.nec.com/carroll98parser.html}, keywords = {parsers evaluation gram_rels}, abstract = {We present a critical overview of the state-of-the-art in parser evaluation methodologies and metrics. A discussion of their relative strengths and weaknesses motivates a new --- and we claim more informative and generally applicable --- technique of measuring parser accuracy, based on the use of grammatical relations. We conclude with some preliminary results of experiments in which we use this new scheme to evaluate a robust parser of English.} }

%0 = inproceedings %A = Carroll, John and Briscoe, Ted and Sanfilippo, Antonio %B = Proc. LREC98 %D = 1998 %T = Parser Evaluation: a Survey and a New Proposal %U = http://citeseer.nj.nec.com/carroll98parser.html