Article published In: International Journal of Corpus Linguistics
Vol. 23:1 (2018) ► pp.28–54
Dependency parsing of learner English
Published online: 31 May 2018
https://doi.org/10.1075/ijcl.16080.hua
https://doi.org/10.1075/ijcl.16080.hua
Abstract
Current syntactic annotation of large-scale learner corpora mainly resorts to “standard parsers” trained on native language data. Understanding how these parsers perform on learner data is important for downstream research and application related to learner language. This study evaluates the performance of multiple standard probabilistic parsers on learner English. Our contributions are three-fold. Firstly, we demonstrate that the common practice of constructing a gold standard – by manually correcting the pre-annotation of a single parser – can introduce bias to parser evaluation. We propose an alternative annotation method which can control for the annotation bias. Secondly, we quantify the influence of learner errors on parsing errors, and identify the learner errors that impact on parsing most. Finally, we compare the performance of the parsers on learner English and native English. Our results have useful implications on how to select a standard parser for learner English.
Article outline
- 1.Introduction
- 2.Automatic dependency annotation of learner language
- 3.Data
- 4.Method
- 4.1Parser evaluation on learner English
- 4.1.1Dependency scheme
- 4.1.2Parsers
- 4.1.3Annotators
- 4.1.4Annotating procedure
- 4.2Investigating the impact of learner errors on parsing
- 4.2.1Learner error scheme
- 4.2.2Annotating learner errors
- 4.2.3Annotating the relation between learner errors and parsing errors
- 4.3Comparison of dependency parsing on learner English and native English
- 4.1Parser evaluation on learner English
- 5.Results
- 5.1Annotation bias on learner English
- 5.1.1Annotation errors on POS tags
- 5.1.2Annotation errors on head indices
- 5.1.3Annotation errors on dependency labels
- 5.2Impact of learner errors on parsing errors
- 5.3Parser performance on learner English and native English
- 5.1Annotation bias on learner English
- 6.Conclusion
References
References (31)
Berzak, Y., Huang, Y., Barbu, A., Korhonen, A., & Katz, B. (2016a). Anchoring and agreement in syntactic annotations. In J. Su (Ed.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 2215–2224). Austin, TX: ACL.
Berzak, Y., Kenney, J., Spadine, C., Wang, J. X., Lam, L., Mori, K. S., Garza, S., & Katz, B. (2016b). Universal dependencies for learner English. In K. Erk & N. A. Smith (Eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 737–746). Berlin: ACL.
Buchholz, S., & Marsi, E. (2006). CoNLL-X shared task on multilingual dependency parsing. In L. Marquez & D. Klein (Eds.), Proceedings of the Tenth Conference on Computational Natural Language Learning (pp. 149–164). New York, NY: ACL.
Cer, D. M., De Marneffe, M. -C., Jurafsky, D., & Manning, C. D. (2010). Parsing to Stanford dependencies: Trade-offs between speed and accuracy. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner & D. Tapias (Eds.), Proceedings of the Seventh International Conference on Language Resources and Evaluation (pp. 1628–1632). Valletta: ELRA.
Charniak, E., & Johnson, M. (2005). Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In K. Knight (Ed.), Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (pp. 173–180). Stroudsburg: ACL.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.
De Marneffe, M. -C., MacCartney, B., & Manning, C. D. (2006). Generating typed dependency parses from phrase structure parses. In N. Calzolari, K. Choukri, A. Gangemi, B. Maegaard, J. Mariani, J. Odijk & D. Tapias (Eds.), Proceedings of the Fifth International Conference on Language Resources and Evaluation (pp. 449–454). Genoa: ELRA.
De Marneffe, M. -C., & Manning, C. D. (2008). Stanford typed dependencies manual (Technical Report). Retrieved from [URL] (last accessed February 2018).
Dickinson, M., & Lee, C. M. (2013). Modifying corpus annotation to support the analysis of learner language. CALICO Journal, 26(3), 545–561.
Dickinson, M., & Ragheb, M. (2009). Dependency annotation for learner corpora. In M. Passarotti, A. Przepiorkowski, S. Raynaud & F. Van Eynde (Eds.), Proceedings of the Eighth Workshop on Treebanks and Linguistic Theories (pp. 59–70). Milan: EDUCatt.
Geertzen, J., Alexopoulou, T., & Korhonen, A. (2013). Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCAMDAT). In R. T. Miller, K. I. Martin, C. M. Eddington, A. Henery, N. M. Miguel, A. Tseng, A. Tuninetti & D. Walter (Eds.), Proceedings of the 31st Second Language Research Forum: Building Bridges Between Disciplines. Somerville: Cascadilla Proceedings Project.
Granger, S., Dagneaux, E., Meunier, F., & Paquot, M. (2009). The International Corpus of Learner English. Version 2. Handbook and CD-ROM. Louvain-la-Neuve: Presses Universitaires de Louvain.
James, C. (2013). Errors in Language Learning and Use: Exploring Error Analysis. New York, NY: Addison Wesley Longman.
Klein, D., & Manning, C. D. (2003a). Accurate unlexicalized parsing. In E. W. Hinrichs & D. Roth (Eds.), Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 1 (pp. 423–430). Sapporo: ACL.
(2003b). Fast exact inference with a factored model for natural language parsing. In S. Becker, S. Thrun, & K. Obermayer (Eds.), Advances in Neural Information Processing Systems 15 (pp. 3–10). Cambridge, MA: MIT Press.
Kong, L., & Smith, N. A. (2014). An empirical comparison of parsing methods for stanford dependencies (arXiv preprint). Retrieved from [URL] (last accessed February 2018).
Korhonen, A. (2002). Semantically motivated subcategorization acquisition. In J. Pentheroudakis, N. Calzolari & A. Wu (Eds.), Proceedings of the ACL-02 Workshop on Unsupervised Lexical Acquisition-Volume 9 (pp. 51–58). Philadelphia, PA: ACL.
Krivanek, J., & Meurers, D. (2011). Comparing rule-based and data-driven dependency parsing of learner language. In K. Gerdes, E. Hajičová & L. Wanner (Eds.), Proceedings of the First International Conference on Dependency Linguistics (Depling 2011) (pp. 310–317). Barcelona: IOS Press.
Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330.
Martins, A. F. T., Almeida, M., & Smith, N. A. (2013). Turning on the Turbo: Fast third-order non-projective Turbo parsers. In H. Schuetze (Ed.), Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL) (pp. 617–622). Sofia: ACL.
Nicholls, D. (2003). The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. In A. Dawn, P. Rayson, A. Wilson & T. McEnery (Eds.), Proceedings of the Corpus Linguistics 2003 Conference (pp. 572–581). Lancaster: UCREL.
Nivre, J., Hall, J., Nilsson, J., Chanev, A., Eryigit, G., Kübler, S., Marinov, S., & Marsi, E. (2007). MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2), 95–135.
Ott, N., & Ziai, R. (2010). Evaluating dependency parsing performance on German learner language. In M. Dickinson, K. Müürisep & M. Passarotti (Eds.), Proceedings of the Ninth International Workshop on Treebanks and Linguistic Theories (pp. 175–186). Tartu: NEALT.
Paquot, M., & Plonsky, L. (2017). Quantitative research methods and study quality in learner corpus research. International Journal of Learner Corpus Research, 3(1), 61–94.
Petrov, S., & Klein, D. (2007). Improved inference for unlexicalized parsing. In B. Carpenter, A. Stent & J. D. Williams (Eds.), Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL) (pp. 404–411). Rochester: ACL.
Ragheb, M., & Dickinson, M. (2011). Avoiding the comparative fallacy in the annotation of learner corpora. In G. Granena, J. Koeth, S. Lee-Ellis, A. Lukyanchenko, G. P. Botana & E. Rhoades (Eds.), Selected Proceedings of the 2010 Second Language Research Forum: Reconsidering SLA Research, Dimensions, and Directions (pp. 114–124). Somerville, MA: Cascadilla Proceedings Project.
(2013). Inter-annotator agreement for dependency annotation of learner language. In J. Tetreault, J. Burstein & C. Leacock (Eds.), Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 169–179). Atlanta, GA: ACL.
Rankin, T. (2015). Review of Clausal Complements in Native and Learner English: A Corpus-Based Study with LINDSEI and VICOLSE
. International Journal of Learner Corpus Research, 1(2), 279–283.
Rosen, A., Hana, J., Štindlová, B., & Feldman, A. (2014). Evaluating and automating the annotation of a learner corpus. Language Resources and Evaluation, 48(1), 65–92.
Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank Project (3rd revision, 2nd printing) (Technical report). Retrieved from [URL] (last accessed February 2018).
Tono, Y., & Díez-Bedmar, M. B. (2014). Focus on learner writing at the beginning and intermediate stages: The ICCI corpus. International Journal of Corpus Linguistics, 19(2), 163–177.
Cited by (39)
Cited by 39 other publications
Geremia, Sara, Thomas Gaillat, Nicolas Ballier & Andrew J. Simpkin
Murakami, Akira
Sato, Masatoshi, Steven L. Thorne, Marije Michel, Theodora Alexopoulou & John Hellermann
Uchida, Satoru & Masashi Negishi
Zhang, Yujie & Lawrence Jun Zhang
Alzahrani, Alaa & Lawrence Jun Zhang
Bannò, Stefano & Marco Matassoni
Eguchi, Masaki & Kristopher Kyle
Granger, Sylviane
2024. From early to future learner corpus research. International Journal of Learner Corpus Research 10:2 ► pp. 247 ff.
Kim, Minjin & Xiaofei Lu
Kyle, Kristopher & Masaki Eguchi
Lestari, Febriana
Ma, Hong, Jinglei Wang & Lianzhen He
Shatz, Itamar, Theodora Alexopoulou & Akira Murakami
Spina, Stefania, Irene Fioravanti, Luciana Forti & Fabio Zanda
Vercellotti, MaryLou & Sean Hall
Xia, Detong, Mark A. Sulzer & Hye K. Pae
Yan, Hengbin & Yinghui Li
Berti, Barbara, Andrea Esuli & Fabrizio Sebastiani
Crossley, Scott & Langdon Holmes
2023. Assessing receptive vocabulary using state‑of‑the‑art natural language processing techniques. Journal of Second Language Studies 6:1 ► pp. 1 ff.
Shatz, Itamar, Theodora Alexopoulou, Akira Murakami & Ramona Bongelli
Du, Xiangtao, Muhammad Afzaal & Hind Al Fadda
Durrant, Philip
Gaillat, Thomas, Andrew Simpkin, Nicolas Ballier, Bernardo Stearns, Annanda Sousa, Manon Bouyé & Manel Zarrouk
Murakami, Akira & Nick C. Ellis
Tan, Yi & Ute Römer
Xia, Detong, Haiyang Ai & Hye K. Pae
Chen, Xiaobin, Theodora Alexopoulou & Ianthi Tsimpli
Huang, Yan, Akira Murakami, Theodora Alexopoulou & Anna Korhonen
2021. Subcategorization frame identification for learner English. International Journal of Corpus Linguistics 26:2 ► pp. 187 ff.
Kyle, Kristopher
2021. Natural language processing for learner corpus research. International Journal of Learner Corpus Research 7:1 ► pp. 1 ff.
Rubin, Rachel
2021. Assessing the impact of automatic dependency annotation on the measurement of phraseological complexity in L2 Dutch. International Journal of Learner Corpus Research 7:1 ► pp. 131 ff.
Sun, Kun & Xiaofei Lu
Sun, Kun & Rong Wang
Gilquin, Gaëtanelle
Shatz, Itamar
2020. Refining and modifying the EFCAMDAT. International Journal of Learner Corpus Research 6:2 ► pp. 220 ff.
Ballier, Nicolas, Thomas Gaillat, Andrew Simpkin, Bernardo Stearns, Manon Bouyé & Manel Zarrouk
[no author supplied]
This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.