Miyao, Y.; Sagae, K. & Tsujii, J.: Towards Framework-Independent Evaluation of Deep Linguistic Parsers. In: Copestake, A. (Hrsg.): Proceedings of the GEAF 2007 Workshop. CSLI Publications, 2007CSLI Studies in Computational Linguistics Online , S. 21 pages
[Volltext]
This paper describes practical issues in the framework-independent evaluation of deep and shallow parsers. We focus on the use of two dependency-based syntactic representation formats in parser evaluation, namely, Carroll et al. (1998)'s Grammatical Relations and de Marneffe et al. (2006)'s Stanford Dependency scheme. Our approach is to convert the output of parsers into these two formats, and measure the accuracy of the resulting converted output. Through the evaluation of an HPSG parser and Penn Treebank phrase structure parsers, we found that mapping between different representation schemes is a non-trivial task that results in lossy conversions that may obscure important differences between different parsing approaches. We discuss sources of disagreements in the representation of syntactic structures in the two dependency-based formats, indicating possible directions for improved framework-independent parser evaluation.

Sagae, K.; Miyao, Y. & Tsujii, J.: HPSG Parsing with Shallow Dependency Constraints. Proc. ACL 2007. 2007, S. 624-631
[Volltext]
We present a novel framework that combines strengths from surface syntactic parsing and deep syntactic parsing to increase deep parsing accuracy, specifically by combining dependency and HPSG parsing. We show that by using surface dependencies to constrain the application of wide-coverage HPSG rules, we can benefit from a number of parsing techniques designed for high-accuracy dependency parsing, while actually performing deep syntactic analysis. Our framework results in a 1.4% absolute improvement over a state-of-the-art approach for wide coverage HPSG parsing.

de Marneffe, M.-C.; MacCartney, B. & Manning, C. D.: Generating Typed Dependency Parses from Phrase Structure Trees. LREC. 2006
[Volltext]

Pyysalo, S.; Ginter, F.; Pahikkala, T.; Boberg, J.; Jarvinen, J. & Salakoski, T.: Evaluation of two dependency parsers on biomedical corpus targeted at protein-protein interactions. In: International Journal of Medical Informatics 75 (2006), Nr. 6, S. 430-442
[Volltext]
We present an evaluation of Link Grammar and Connexor Machinese Syntax, two major broad-coverage dependency parsers, on a custom hand-annotated corpus consisting of sentences regarding protein-protein interactions. In the evaluation, we apply the notion of an interaction subgraph, which is the subgraph of a dependency graph expressing a protein-protein interaction. We measure the performance of the parsers for recovery of individual dependencies, fully correct parses, and interaction subgraphs. For Link Grammar, an open system that can be inspected in detail, we further perform a comprehensive failure analysis, report specific causes of error, and suggest potential modifications to the grammar. We find that both parsers perform worse on biomedical English than previously reported on general English. While Connexor Machinese Syntax significantly outperforms Link Grammar, the failure analysis suggests specific ways in which the latter could be modified for better performance in the domain.

Miyao, Y.; Ninomiya, T. & Tsujii, J.: Corpus-oriented Grammar Development for Acquiring a Head-driven Phrase Structure Grammar from the Penn Treebank. Proceedings of IJCNLP-04. 8 pages: 2004
[Volltext]
This paper describes a method of semi-automatically acquiring an English HPSG grammar from the Penn Treebank. First, heuristic rules are employed to annotate the treebank with partially-specified derivation trees. Lexical entries are automatically extracted from the annotated corpus by inversely applying schemata to partially-specified derivation trees.

Mollá, D. & Hutchinson, B.: Intrinsic versus Extrinsic Evaluations of Parsing Systems. Proc. European Association for Computational Linguistics (EACL), workshop on Evaluation Initiatives in Natural Language Processing. Budapest: ACL, 2003, S. 43-50

A wide range of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (intrinsic evaluations), and only in a few cases has the effect of different parsers in real applications been measured (extrinsic evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the intrinsic evaluation, the accuracy of the parsers is compared independently by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the extrinsic evaluation, the parsers' impact in a practical application is compared within the context of answer extraction. The differences in the results are significant.

Schneider, G.: Extracting and Using Trace-Free Functional Dependencies from the Penn Treebank to Reduce Parsing Complexity. Proc. Workshop on Treebanks and Linguistic Theories (TLT 2003). 2003
[Volltext]
Many extensions to text-based, data-intensive knowledge management approaches, such as Information Retrieval or Data Mining, focus on integrating the impressive recent advances in language technology. For this, they need fast, robust parsers that deliver linguistic data which is meaningful for the subsequent processing stages. This paper introduces such a parsing system. Its output is a hierarchical structure of syntactic relations, functional dependency structures ...

Mollá, D. & Hutchinson, B.: In Vitro and In Vivo Evaluations of Parsing Systems Within the Context of Answer Extraction. , 2002

A wide variety of parser and/or grammar evaluation methods have been reported in the literature. However, in most cases these evaluations take the parsers independently (in vitro evaluations), and only in a few cases has the effect of different parsers in real applications been measured (in vivo evaluations). This paper compares two evaluations of the Link Grammar parser and the Conexor Functional Dependency Grammar parser. The parsing systems, despite both being dependency-based, return different types of dependencies, making a direct comparison impossible. In the first evaluation, the accuracy of the parsers is compared in vitro by converting the dependencies into grammatical relations and using the methodology of Carroll:1998 for parser comparison. In the second evaluation, the parsers' impact in a practical application is compared in vivo within the context of answer extraction. The differences in the results are significant and raise questions on the usefulness of purely in vitro evaluations.

Oy, C.: Conexor Functional Dependency Grammar 3.7 -- User's Manual. , 2002

Scarlett, E.: An Evaluation of a Rule-Based Parser of English Sentences. University of Ottawa, 2000
[Volltext]
... The thesis argues that a test suite for a broad coverage natural language parser must necessarily be systematic, broad in its coverage of phenomena tested, and corpus-like in its coverage of phenomenon interaction. A test suite of example sentences extracted from Quirk et al.'s comprehensive English grammar is proposed, and the results of evaluating DIPETT on that suite are compared with the evaluation results on a publicly available test suite, TSNLP (Test Suites for Natural Language Processing).

Scarlett, E. & Szpkowicz, S.: The Power of the TSNLP: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser. Proc. Thirteenth Canadian Conference on Artificial Intelligence. Montreal: 2000
[Volltext]
We show a diagnostic evaluation of DIPETT, a broad-coverage parser of English sentences. We consider the TSNLP suite as a diagnostic tool, and propose an alternative broader-coverage test suite of test sentences extracted from Quirk et al. We compare the diagnostic effectiveness of the two suites, and draw a few general conclusions. The evaluation results were used to make significant improvements to DIPETT.

Henderson, J. C. & Brill, E.: Exploiting Diversity in Natural Language Processing: Combining Parsers. Proc. EMNLP-99. College Park, Maryland: 1999, S. 187-194
[Volltext]
Three state-of-the-art statistical parsers are combined to produce more accurate parses, as well as new bounds on achievable Treebank parsing accuracy. Two general approaches are presented and two combination techniques are described for each approach. Both parametric and non-parametric models are explored. The resulting parsers surpass the best previously published performance results for the Penn Treebank.

Roark, B. & Johnson, M.: Efficient Probabilistic Top-down and Left-corner Parsing. Proc. ACL'99. 1999, S. 421-428
[Volltext]
This paper examines efficient predictive broad-coverage parsing without dynamic programming. In contrast to bottom-up methods, depth-first top-down parsing produces partial parses that are fully connected trees spanning the entire left context, from which any kind of non-local dependency or partial semantic interpretation can in principle be read. We contrast two predictive parsing approaches, top-down and left-corner parsing, and find both to be viable. In addition, we find that enhancement with non-local information not only improves parser accuracy, but also substantially improves the search efficiency.

Järvinen, T. & Tapanainen, P.: Towards an Implementable Dependency Grammar. Proc. of COLING-ACL. Workshop: Processing of Dependency Grammars. 1998

Lin, D.: Dependency-based Evaluation of MINIPAR. Proc. Workshop on the Evaluation of Parsing Systems. 1998Granada

Schneider, G.: Functional Non-Projective Parsing with Dependency Grammar (DG). , 1998

Järvinen, T. & Tapanainen, P.: A Dependency Parser for English. Helsinki, 1997
[Volltext]

Tapanainen, P. & Järvinen, T.: A Non-Projective Dependency Parser. Proc. ANLP-97. 1997
[Volltext]

Kay, M.: Chart Generation. Proc. 34th Annual Meeting of the ACL. Santa Cruz, CA: 1996, S. 200-204

Charts constitute a natural uniform architecture for parsing and generation provided string position is replaced by a notion more appropriate to logical forms and that measures are taken to curtail generation paths containing semantically incomplete phrases.

Tomita, M.: An Efficient Augmented-Context-Free Parsing Algorithm. In: Computational Linguistics 12 (1987), Nr. 1-2, S. 31-46