TY - CONF AU - Miyao, Yusuke AU - Sagae, Kenji AU - Tsujii, Jun'ichi A2 - Copestake, Ann T1 - Towards Framework-Independent Evaluation of Deep Linguistic Parsers T2 - Proceedings of the GEAF 2007 Workshop PB - CSLI Publications CY - PY - 2007/ M2 - IS - SP - EP - UR - http://www.cs.cmu.edu/~sagae/docs/geaf07miyaoetal.pdf M3 - KW - parsers evaluation L1 - SN - N1 - N1 - AB - This paper describes practical issues in the framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependency-based syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)'s Grammatical Relations and de Marneffe et al. (2006)'s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation. ER - TY - CONF AU - Sagae, Kenji AU - Miyao, Yusuke AU - Tsujii, Jun'ichi A2 - T1 - HPSG Parsing with Shallow Dependency Constraints T2 - Proc. ACL 2007 PB - CY - PY - 2007/ M2 - IS - SP - 624 EP - 631 UR - http://www-tsujii.is.s.u-tokyo.ac.jp/enju/ M3 - KW - HPSG parsers L1 - SN - N1 - N1 - AB - We present a novel framework that combines strengths from surface syntactic parsing and deep syntactic parsing to increase deep parsing accuracy, specifically by combining dependency and HPSG parsing. We show that by using surface dependencies to constrain the application of wide-coverage HPSG rules, we can benefit from a number of parsing techniques designed for high-accuracy dependency parsing, while actually performing deep syntactic analysis. Our framework results in a 1.4% absolute improvement over a state-of-the-art approach for wide coverage HPSG parsing. ER - TY - CONF AU - de Marneffe, Marie-Catherine AU - MacCartney, Bill AU - Manning, Christopher D. A2 - T1 - Generating Typed Dependency Parses from Phrase Structure Trees T2 - LREC PB - CY - PY - 2006/ M2 - IS - SP - EP - UR - http://nlp.stanford.edu/pubs/LREC06_dependencies.pdf M3 - KW - dependencies constituency parsers L1 - SN - N1 - literature-log.bib 1/21/2009 N1 - AB - ER - TY - JOUR AU - Pyysalo, Sampo AU - Ginter, Filip AU - Pahikkala, Tapio AU - Boberg, Jorma AU - Jarvinen, Jouni AU - Salakoski, Tapio T1 - Evaluation of two dependency parsers on biomedical corpus targeted at protein-protein interactions JO - International Journal of Medical Informatics PY - 2006/06 VL - 75 IS - 6 SP - 430 EP - 442 UR - http://dx.doi.org/10.1016/j.ijmedinf.2005.06.009 M3 - 10.1016/j.ijmedinf.2005.06.009 KW - parsers biomedicine evaluation L1 - SN - N1 - N1 - AB - We present an evaluation of Link Grammar and Connexor Machinese Syntax, two major broad-coverage dependency parsers, on a custom hand-annotated corpus consisting of sentences regarding protein-protein interactions. In the evaluation, we apply the notion of an interaction subgraph, which is the subgraph of a dependency graph expressing a protein-protein interaction. We measure the performance of the parsers for recovery of individual dependencies, fully correct parses, and interaction subgraphs. For Link Grammar, an open system that can be inspected in detail, we further perform a comprehensive failure analysis, report specific causes of error, and suggest potential modifications to the grammar. We find that both parsers perform worse on biomedical English than previously reported on general English. While Connexor Machinese Syntax significantly outperforms Link Grammar, the failure analysis suggests specific ways in which the latter could be modified for better performance in the domain. ER - TY - CONF AU - Miyao, Yusuke AU - Ninomiya, Takashi AU - Tsujii, Jun'ichi A2 - T1 - Corpus-oriented Grammar Development for Acquiring a Head-driven Phrase Structure Grammar from the Penn Treebank T2 - Proceedings of IJCNLP-04 PB - CY - 8 pages PY - 2004/ M2 - IS - SP - EP - UR - http://www-tsujii.is.s.u-tokyo.ac.jp/~yusuke/paper/ijcnlp04.pdf M3 - KW - HPSG parsers L1 - SN - N1 - N1 - AB - This paper describes a method of semi-automatically acquiring an English HPSG grammar from the Penn Treebank. First, heuristic rules are employed to annotate the treebank with partially-specified derivation trees. Lexical entries are automatically extracted from the annotated corpus by inversely applying schemata to partially-specified derivation trees. ER - TY - CONF AU - Mollá, Diego AU - Hutchinson, Ben A2 - T1 - Intrinsic versus Extrinsic Evaluations of Parsing Systems T2 - Proc. European Association for Computational Linguistics (EACL), workshop on Evaluation Initiatives in Natural Language Processing PB - ACL CY - Budapest PY - 2003/04 M2 - IS - SP - 43 EP - 50 UR - M3 - KW - parsers evaluation AnswerFinder gram_rels molla_publication L1 - SN - N1 - N1 - AB - A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant. ER - TY - CONF AU - Schneider, Gerold A2 - T1 - Extracting and Using Trace-Free Functional Dependencies from the Penn Treebank to Reduce Parsing Complexity T2 - Proc. Workshop on Treebanks and Linguistic Theories (TLT 2003) PB - CY - PY - 2003/ M2 - IS - SP - EP - UR - http://w3.msi.vxu.se/\~{}rics/TLT2003/ M3 - KW - dependencies parsers L1 - SN - N1 - N1 - AB - Many extensions to text-based, data-intensive knowledge management approaches, such as Information Retrieval or Data Mining, focus on integrating the impressive recent advances in language technology. For this, they need fast, robust parsers that deliver linguistic data which is meaningful for the subsequent processing stages. This paper introduces such a parsing system. Its output is a hierarchical structure of syntactic relations, functional dependency structures ... ER - TY - UNPB AU - Mollá, Diego AU - Hutchinson, Ben A2 - T1 - In Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction PY - 2002/ SP - EP - UR - M3 - KW - AnswerFinder parsers evaluation gram_rels molla_publication L1 - N1 - N1 - AB - A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (in vitro evaluations), and only in a few cases has the effect of different parsers in real applications been measured (in vivo evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared in vitro by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared in vivo within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely in vitro evaluations. ER - TY - RPRT AU - Oy, Conexor A2 - T1 - Conexor Functional Dependency Grammar 3.7 --- User's Manual PB - Conexor Oy AD - PY - 2002/ VL - IS - SP - EP - UR - M3 - KW - parsers DG L1 - N1 - N1 - N1 - AB - ER - TY - THES AU - Scarlett, Elizabeth T1 - An Evaluation of a Rule-Based Parser of English Sentences PY - 2000/ PB - University of Ottawa SP - EP - UR - http://www.site.uottawa.ca/\~{}scarlett/ M3 - KW - parsers evaluation L1 - N1 - N1 - AB - ... The thesis argues that a test suite for a broad coverage natural language parser must necessarily be systematic, broad in its coverage of phenomena tested, and corpus-like in its coverage of phenomenon interaction. A test suite of example sentences extracted from Quirk et al.'s comprehensive English grammar is proposed, and the results of evaluating DIPETT on that suite are compared with the evaluation results on a publicly available test suite, TSNLP (Test Suites for Natural Language Processing). ER - TY - CONF AU - Scarlett, Elizabeth AU - Szpkowicz, Stan A2 - T1 - The Power of the TSNLP: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser T2 - Proc. Thirteenth Canadian Conference on Artificial Intelligence PB - CY - Montreal PY - 2000/ M2 - IS - SP - EP - UR - http://www.site.uottawa.ca/\~{}scarlett/ M3 - KW - parsers evaluation L1 - SN - N1 - N1 - AB - We show a diagnostic evaluation of DIPETT, a broad-coverage parser of English sentences. We consider the TSNLP suite as a diagnostic tool, and propose an alternative broader-coverage test suite of test sentences extracted from Quirk et al. We compare the diagnostic effectiveness of the two suites, and draw a few general conclusions. The evaluation results were used to make significant improvements to DIPETT. ER - TY - CONF AU - Henderson, John C. AU - Brill, Eric A2 - T1 - Exploiting Diversity in Natural Language Processing: Combining Parsers T2 - Proc. EMNLP-99 PB - CY - College Park, Maryland PY - 1999/ M2 - IS - SP - 187 EP - 194 UR - http://xxx.lanl.gov/abs/cs.CL/0006003 M3 - KW - parsers L1 - SN - N1 - N1 - AB - Three state-of-the-art statistical parsers are combined to produce more accurate parses, as well as new bounds on achievable Treebank parsing accuracy. Two general approaches are presented and two combination techniques are described for each approach. Both parametric and non-parametric models are explored. The resulting parsers surpass the best previously published performance results for the Penn Treebank. ER - TY - CONF AU - Roark, Brian AU - Johnson, Mark A2 - T1 - Efficient Probabilistic Top-down and Left-corner Parsing T2 - Proc. ACL'99 PB - CY - PY - 1999/ M2 - IS - SP - 421 EP - 428 UR - http://arXiv.org/abs/cs/0008017 M3 - KW - parsers statistic_nlp L1 - SN - N1 - N1 - AB - This paper examines efficient predictive broad-coverage parsing without dynamic programming. In contrast to bottom-up methods, depth-first top-down parsing produces partial parses that are fully connected trees spanning the entire left context, from which any kind of non-local dependency or partial semantic interpretation can in principle be read. We contrast two predictive parsing approaches, top-down and left-corner parsing, and find both to be viable. In addition, we find that enhancement with non-local information not only improves parser accuracy, but also substantially improves the search efficiency. ER - TY - CONF AU - Järvinen, Timo AU - Tapanainen, Pasi A2 - T1 - Towards an Implementable Dependency Grammar T2 - Proc. of COLING-ACL. Workshop: Processing of Dependency Grammars PB - CY - PY - 1998/ M2 - IS - SP - EP - UR - M3 - KW - DG parsers L1 - SN - N1 - N1 - AB - ER - TY - CONF AU - Lin, Dekang A2 - T1 - Dependency-based Evaluation of MINIPAR T2 - Proc. Workshop on the Evaluation of Parsing Systems PB - CY - PY - 1998/ M2 - IS - SP - EP - UR - M3 - KW - parsers DG L1 - SN - N1 - N1 - AB - ER - TY - GEN AU - Schneider, Gerold A2 - T1 - Functional Non-Projective Parsing with Dependency Grammar (DG) JO - PB - AD - PY - 1998/november VL - IS - SP - EP - UR - M3 - KW - DG parsers L1 - N1 - N1 - AB - ER - TY - RPRT AU - Järvinen, Timo AU - Tapanainen, Pasi A2 - T1 - A Dependency Parser for English PB - Department of Linguistics, University of Helsinki AD - Helsinki PY - 1997/ VL - IS - TR-1 SP - EP - UR - http://www.ling.helsinki.fi/\~{}tapanain/dg/doc/TR-1/ M3 - KW - DG parsers L1 - N1 - N1 - N1 - AB - ER - TY - CONF AU - Tapanainen, Pasi AU - Järvinen, Timo A2 - T1 - A Non-Projective Dependency Parser T2 - Proc. ANLP-97 PB - CY - PY - 1997/ M2 - IS - SP - EP - UR - http://www.conexor.fi/anlp97/anlp97.html M3 - KW - DG parsers L1 - SN - N1 - N1 - AB - ER - TY - CONF AU - Kay, Martin A2 - T1 - Chart Generation T2 - Proc. 34th Annual Meeting of the ACL PB - CY - Santa Cruz, CA PY - 1996/ M2 - IS - SP - 200 EP - 204 UR - M3 - KW - parsers flat_semantics generation L1 - SN - N1 - N1 - AB - Charts constitute a natural uniform architecture for parsing and generation provided string position is replaced by a notion more appropriate to logical forms and that measures are taken to curtail generation paths containing semantically incomplete phrases. ER - TY - JOUR AU - Tomita, Masaru T1 - An Efficient Augmented-Context-Free Parsing Algorithm JO - Computational Linguistics PY - 1987/ VL - 12 IS - 1-2 SP - 31 EP - 46 UR - M3 - KW - parsers L1 - SN - N1 - N1 - AB - ER -