Miyao,Yusuke Sagae,Kenji Tsujii,Jun'ichi Towards Framework-Independent Evaluation of Deep Linguistic Parsers CSLI Publications 21 pages 2007 This paper describes practical issues in the framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependency-based syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)'s Grammatical Relations and de Marneffe et al. (2006)'s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation. Sagae,Kenji Miyao,Yusuke Tsujii,Jun'ichi HPSG Parsing with Shallow Dependency Constraints 624-631 2007 We present a novel framework that combines strengths from surface syntactic parsing and deep syntactic parsing to increase deep parsing accuracy, specifically by combining dependency and HPSG parsing. We show that by using surface dependencies to constrain the application of wide-coverage HPSG rules, we can benefit from a number of parsing techniques designed for high-accuracy dependency parsing, while actually performing deep syntactic analysis. Our framework results in a 1.4% absolute improvement over a state-of-the-art approach for wide coverage HPSG parsing. deMarneffe,Marie-Catherine MacCartney,Bill Manning,ChristopherD. Generating Typed Dependency Parses from Phrase Structure Trees 2006 Pyysalo,Sampo Ginter,Filip Pahikkala,Tapio Boberg,Jorma Jarvinen,Jouni Salakoski,Tapio Evaluation of two dependency parsers on biomedical corpus targeted at protein-protein interactions International Journal of Medical Informatics 75 430–442 2006 We present an evaluation of Link Grammar and Connexor Machinese Syntax, two major broad-coverage dependency parsers, on a custom hand-annotated corpus consisting of sentences regarding protein-protein interactions. In the evaluation, we apply the notion of an interaction subgraph, which is the subgraph of a dependency graph expressing a protein-protein interaction. We measure the performance of the parsers for recovery of individual dependencies, fully correct parses, and interaction subgraphs. For Link Grammar, an open system that can be inspected in detail, we further perform a comprehensive failure analysis, report specific causes of error, and suggest potential modifications to the grammar. We find that both parsers perform worse on biomedical English than previously reported on general English. While Connexor Machinese Syntax significantly outperforms Link Grammar, the failure analysis suggests specific ways in which the latter could be modified for better performance in the domain. Miyao,Yusuke Ninomiya,Takashi Tsujii,Jun'ichi Corpus-oriented Grammar Development for Acquiring a Head-driven Phrase Structure Grammar from the Penn Treebank 2004 This paper describes a method of semi-automatically acquiring an English HPSG grammar from the Penn Treebank. First, heuristic rules are employed to annotate the treebank with partially-specified derivation trees. Lexical entries are automatically extracted from the annotated corpus by inversely applying schemata to partially-specified derivation trees. Mollá,Diego Hutchinson,Ben Intrinsic versus Extrinsic Evaluations of Parsing Systems ACL 43-50 2003 A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of :1998 for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant. Schneider,Gerold Extracting and Using Trace-Free Functional Dependencies from the Penn Treebank to Reduce Parsing Complexity 2003 Many extensions to text-based, data-intensive knowledge management approaches, such as Information Retrieval or Data Mining, focus on integrating the impressive recent advances in language technology. For this, they need fast, robust parsers that deliver linguistic data which is meaningful for the subsequent processing stages. This paper introduces such a parsing system. Its output is a hierarchical structure of syntactic relations, functional dependency structures ... Mollá,Diego Hutchinson,Ben In Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction 2002 A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently ( vitro evaluations), and only in a few cases has the effect of different parsers in real applications been measured ( vivo evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared vitro by converting the dependencies into grammatical relations and using the methodology of :1998 for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared vivo within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely vitro evaluations. Oy,Conexor Conexor Functional Dependency Grammar 3.7 –- User's Manual 2002 Scarlett,Elizabeth An Evaluation of a Rule-Based Parser of English Sentences 2000 ... The thesis argues that a test suite for a broad coverage natural language parser must necessarily be systematic, broad in its coverage of phenomena tested, and corpus-like in its coverage of phenomenon interaction. A test suite of example sentences extracted from Quirk et al.'s comprehensive English grammar is proposed, and the results of evaluating DIPETT on that suite are compared with the evaluation results on a publicly available test suite, TSNLP (Test Suites for Natural Language Processing). Scarlett,Elizabeth Szpkowicz,Stan The Power of the TSNLP: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser 2000 We show a diagnostic evaluation of DIPETT, a broad-coverage parser of English sentences. We consider the TSNLP suite as a diagnostic tool, and propose an alternative broader-coverage test suite of test sentences extracted from Quirk et al. We compare the diagnostic effectiveness of the two suites, and draw a few general conclusions. The evaluation results were used to make significant improvements to DIPETT. Henderson,JohnC. Brill,Eric Exploiting Diversity in Natural Language Processing: Combining Parsers 187-194 1999 Three state-of-the-art statistical parsers are combined to produce more accurate parses, as well as new bounds on achievable Treebank parsing accuracy. Two general approaches are presented and two combination techniques are described for each approach. Both parametric and non-parametric models are explored. The resulting parsers surpass the best previously published performance results for the Penn Treebank. Roark,Brian Johnson,Mark Efficient Probabilistic Top-down and Left-corner Parsing 421-428 1999 This paper examines efficient predictive broad-coverage parsing without dynamic programming. In contrast to bottom-up methods, depth-first top-down parsing produces partial parses that are fully connected trees spanning the entire left context, from which any kind of non-local dependency or partial semantic interpretation can in principle be read. We contrast two predictive parsing approaches, top-down and left-corner parsing, and find both to be viable. In addition, we find that enhancement with non-local information not only improves parser accuracy, but also substantially improves the search efficiency. Järvinen,Timo Tapanainen,Pasi Towards an Implementable Dependency Grammar 1998 Lin,Dekang Dependency-based Evaluation of MINIPAR 1998 Schneider,Gerold Functional Non-Projective Parsing with Dependency Grammar (DG) 1998 Järvinen,Timo Tapanainen,Pasi A Dependency Parser for English 1997 Tapanainen,Pasi Järvinen,Timo A Non-Projective Dependency Parser 1997 Kay,Martin Chart Generation 200-204 1996 Charts constitute a natural uniform architecture for parsing and generation provided string position is replaced by a notion more appropriate to logical forms and that measures are taken to curtail generation paths containing semantically incomplete phrases. Tomita,Masaru An Efficient Augmented-Context-Free Parsing Algorithm Computational Linguistics 12 31-46 1987