Cover not available

Article published In: International Journal of Corpus Linguistics
Vol. 28:2 (2023) ► pp.232–262

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

A comparison of automated and manual analyses of syntactic complexity in L2 English writing

ORCiD logo with link | Vietnam National University, Hanoi | University of Brussels (VUB)
ORCiD logo with link | University of Brussels (VUB)
Published online: 17 October 2022
DOI logo with link
https://doi.org/10.1075/ijcl.20181.cha
Abstract
Automated tools for syntactic complexity measurement are increasingly used for analyzing various kinds of second language corpora, even though these tools were originally developed and tested for texts produced by advanced learners. This study investigates the reliability of automated complexity measurement for beginner and lower-intermediate L2 English data by comparing manual and automated analyses of a corpus of 80 texts written by Dutch-speaking learners. Our quantitative and qualitative analyses reveal that the reliability of automated complexity measurement is substantially affected by learner errors, parser errors, and Tregex pattern undergeneration. We also demonstrate the importance of aligning the definitions of analytical units between the computational tool and human annotators. In order to enhance the reliability of automated analyses, it is recommended that certain modifications are made to the system, and non-advanced L2 English data are preprocessed prior to automated analyses.
Article outline
  • 1.Introduction
  • 2.L2SCA in L2 English research
    • 2.1L2SCA system information
    • 2.2L2SCA system reliability
    • 2.3Studies that used L2SCA
  • 3.Methodology
    • 3.1Corpus
    • 3.2Syntactic complexity measures
    • 3.3Corpus analysis
    • 3.4Comparison of manual and automated analyses
      • 3.4.1System-annotator agreement in unit and structure identification (RQ 1a)
      • 3.4.2Sources of system-annotator disagreement (RQ 1b)
      • 3.4.3System reliability in syntactic complexity scoring (RQ 2)
  • 4.Results
    • 4.1RQ1a: Unit and structure identification
    • 4.2RQ 1b: Sources of system-annotator disagreement
      • 4.2.1Words
      • 4.2.2Clausal and supra-clausal structures
        • Tregex pattern undergeneration
        • Parsing errors
        • Learner errors
    • 4.3RQ 2a: System reliability in syntactic complexity scoring
    • 4.4RQ 2b: Impact of learner errors and the use of broader definitions of clauses and dependent clauses
      • 4.4.1Impact of learner errors
      • 4.4.2Impact of broader clause and dependent clause definitions
  • 5.Discussion
  • 6.Conclusion
  • References
References (51)
References
Bulté, B. (2013). The Development of Complexity in Second Language Acquisition: A Dynamic Systems Approach [Unpublished doctoral dissertation]. Vrije Universiteit Brussel.
Bulté, B., & Housen, A. (2012). Defining and operationalising L2 complexity. In A. Housen, F. Kuiken, & I. Vedder (Eds.), Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA (pp. 21–46). John Benjamins. Google Scholar logo with link to Google Scholar
(2014). Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 261, 42–65. Google Scholar logo with link to Google Scholar
(2018). Syntactic complexity in L2 writing: Individual pathways and emerging group trends. International Journal of Applied Linguistics, 28(1), 147–164. Google Scholar logo with link to Google Scholar
Bulté, B., & Roothooft, H. (2020). Investigating the interrelationship between rated L2 proficiency and linguistic complexity in L2 speech. System, 911, 102246. Google Scholar logo with link to Google Scholar
Abney, S. P. (1987). The English Noun Phrase in its Sentential Aspect [Doctoral dissertation, Massachusetts Institute of Technology]. DSpace@MIT. [URL]
Ai, H., & Lu, X. (2013). A corpus-based comparison of syntactic complexity in NNS and NS university students’ writing. In A. Díaz-Negrillo, N. Ballier, & P. Thompson (Eds.), Automatic Treatment and Analysis of Learner Corpus Data (pp. 249–264). John Benjamins. Google Scholar logo with link to Google Scholar
Bi, P., & Jiang, J. (2020). Syntactic complexity in assessing young adolescent EFL learners’ writings: Syntactic elaboration and diversity. System, 911, 102248. Google Scholar logo with link to Google Scholar
Biber, D. (1988). Variation Across Speech and Writing. Cambridge University Press. Google Scholar logo with link to Google Scholar
Cambridge University Press & Assessment. (n.d.). Cambridge learner corpus – error codes. [URL]
Casal, J. E., & Lee, J. J. (2019). Syntactic complexity and writing quality in assessed first-year L2 writing. Journal of Second Language Writing, 441, 51–62. Google Scholar logo with link to Google Scholar
Chen, X. B., & Meurers, D. (2016). CTAP: A Web-Based Tool Supporting Automatic Complexity Analysis. Proceedings of The Workshop on Computational Linguistics for Linguistic Complexity (pp. 113–119). Association for Computational Linguistics. [URL]
Choi, J. D., Tetreault, J., & Stent, A. (2015). It depends: Dependency parser comparison using a web-based evaluation tool. In C. Zong & M. Strube (Eds.), Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (pp. 387–396). Association for Computational Linguistics. [URL].
Cooper, T. C. (1976). Measuring written syntactic patterns of second language learners of German. The Journal of Educational Research, 69(5), 176–183. Google Scholar logo with link to Google Scholar
De Clercq, B., & Housen, A. (2017). A cross-linguistic perspective on syntactic complexity in L2 Development: Syntactic elaboration and diversity. The Modern Language Journal, 101(2), 315–334. Google Scholar logo with link to Google Scholar
Gaillat, T., & Ballier, N. (2019). Prototype de feedback visuel des productions écrites d’apprenants francophones de l’anglais sous Moodle [Prototype of visual feedback for written productions of French-speaking learners of English on Moodle]. In Actes de la conférence EIAH2019. Association des Technologies de l’Information pour l’Education et la Formation. [URL]
Granger, S., Dagneaux, E., Meunier, F., & Paquot, M. (2009). International Corpus of Learner English (Version 2.0). Presses universitaires de Louvain.Google Scholar logo with link to Google Scholar
Huibregtse, I., Admiraal, W., & Meara, P. (2002). Scores on a yes-no vocabulary test: Correction for guessing and response style. Language Testing, 19(3), 227–245. Google Scholar logo with link to Google Scholar
Hunt, K. W. (1965). Grammatical Structures Written at Three Grade Levels. National Council of Teachers of English.Google Scholar logo with link to Google Scholar
Hwang, H., Jung, H., & Kim, H. (2020). Effects of written versus spoken production modalities on syntactic complexity measures in beginning-level child EFL learners. The Modern Language Journal, 104(1), 267–283. Google Scholar logo with link to Google Scholar
Jiang, J., Bi, P., & Liu, H. (2019). Syntactic complexity development in the writings of EFL learners: Insights from a dependency syntactically-annotated corpus. Journal of Second Language Writing, 461, 100666. Google Scholar logo with link to Google Scholar
Kameen, P. T. (1979). Syntactic skill and ESL writing quality. In C. Yorio, K. Perkins, & J. Schachter (Eds.), On TESOL ’79: The Learner in Focus (pp. 343–364). TESOL.Google Scholar logo with link to Google Scholar
Khushik, G. A., & Huhta, A. (2019). Investigating syntactic complexity in EFL learners’ writing across Common European Framework of Reference levels A1, A2, and B1. Applied Linguistics, 41(4), 506–532. Google Scholar logo with link to Google Scholar
Klein, D., & Manning, C. D. (2003). Fast exact inference with a factored model for natural language parsing. In S. Beker, S. Thrun, & K. Obermayer (Eds.), Advances in Neural Information Processing Systems 151 (pp. 3–10). MIT Press.Google Scholar logo with link to Google Scholar
Kummerfeld, J. K., Hall, D., Curran, J., & Klein, D. (2012). Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In J. Tsujii, J. Henderson, & M. Paşca (Eds.), The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 1048–1059). [URL]
Kyle, K. (2016). Measuring Syntactic Development in L2 Writing: Fine Grained Indices of Syntactic Complexity and Usage-based Indices of Syntactic Sophistication [Doctoral dissertation, Georgia State University]. ScholarWorks @ Georgia State University. [URL]
(2021). Natural language processing for learner corpus research. International Journal of Learner Corpus Research, 7(1), 1–16. Google Scholar logo with link to Google Scholar
Larsson, T., & Kaatari, H. (2020). Syntactic complexity across registers: Investigating (in)formality in second-language writing. Journal of English for Academic Purposes, 451, 100850. Google Scholar logo with link to Google Scholar
Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474–496. Google Scholar logo with link to Google Scholar
(2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45(1), 36–62. Google Scholar logo with link to Google Scholar
(2014). Computational Methods for Corpus Annotation and Analysis. Springer. Google Scholar logo with link to Google Scholar
(2017). Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Language Testing, 34(4), 493–511. Google Scholar logo with link to Google Scholar
Lu, X., & Ai, H. (2015). Syntactic complexity in college-level English writing: Differences among writers with diverse L1 backgrounds. Journal of Second Language Writing, 291, 16–27. Google Scholar logo with link to Google Scholar
Lu, X., Casal, J. E., & Liu, Y. (2020). The rhetorical functions of syntactically complex sentences in social science research article introductions. Journal of English for Academic Purposes, 441, 100832. Google Scholar logo with link to Google Scholar
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014). The Stanford CoreNLP Natural Language Processing Toolkit. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 55–60). Association for Computational Linguistics. [URL].
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic features of writing quality. Written Communication, 27(1), 57–86. Google Scholar logo with link to Google Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation Of Text And Discourse With Coh-metrix. Cambridge University Press. Google Scholar logo with link to Google Scholar
Meurers, D., & Dickinson, M. (2017). Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67(S1), 66–95. Google Scholar logo with link to Google Scholar
Nicholls, D. (2003). The Cambridge Learner Corpus: Error coding and analysis for lexicography and ELT. Proceedings of the Corpus Linguistics 2003 conference, 572–581.Google Scholar logo with link to Google Scholar
Ortega, L. (2003). Syntactic complexity measures and their relationship to L2 Proficiency: A research synthesis of college-level L2 Writing. Applied Linguistics, 24(4), 492–518. Google Scholar logo with link to Google Scholar
Pallotti, G. (2015). A simple view of linguistic complexity. Second Language Research, 31(1), 117–134. Google Scholar logo with link to Google Scholar
Polio, C. G. (1997). Measures of linguistic accuracy in second language writing research. Language Learning, 47(1), 101–143. Google Scholar logo with link to Google Scholar
Polio, C., & Yoon, H. (2018). The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics, 28(1), 165–188. Google Scholar logo with link to Google Scholar
R Core Team. (2020). R: A language and environment for statistical computing (Version 4.0.0) [Computer software]. R Foundation for Statistical Computing. [URL]
Spoelman, M., & Verspoor, M. (2010). Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics, 31(4), 532–553. Google Scholar logo with link to Google Scholar
Stanford Natural Language Processing Group. (n.d.). Stanford Parser FAQ. Retrieved November 10, 2020, from [URL]
Verspoor, M., de Bot, C., & Xu, X. (2015). The effects of English bilingual education in the Netherlands. Journal of Immersion and Content-Based Language Education, 3(1), 4–27. Google Scholar logo with link to Google Scholar
Verspoor, M., Schmid, M. S., & Xu, X. (2012). A dynamic usage based perspective on L2 writing. Journal of Second Language Writing, 21(3), 239–263. Google Scholar logo with link to Google Scholar
Vyatkina, N. (2013). Specific syntactic complexity: Developmental profiling of individuals based on an annotated learner corpus. The Modern Language Journal, 97(S1), 11–30. Google Scholar logo with link to Google Scholar
Walter, T. (2017). Measuring Syntactic Complexity in the Academic Writing of English Students at the University of Vienna. [Doctoral dissertation, Universität Wien]. u:theses Universität Wien. Google Scholar logo with link to Google Scholar
Wu, X., Mauranen, A., & Lei, L. (2020). Syntactic complexity in English as a Lingua Franca academic writing. Journal of English for Academic Purposes, 431, 100798. Google Scholar logo with link to Google Scholar
Cited by (9)

Cited by nine other publications

Bulté, Bram, Alex Housen & Gabriele Pallotti
2025. Complexity and Difficulty in Second Language Acquisition: A Theoretical and Methodological Overview. Language Learning 75:2 ► pp. 533 ff. DOI logo
Nguyen, Long Quoc & Duy Van Vu
2025. Enhancing EFL Learners’ Expository Writing: Evaluating the Impact of Model Texts as a Feedback Instrument on Draft Quality in Rewritten and New Texts. English Teaching & Learning 49:4 ► pp. 759 ff. DOI logo
Alzahrani, Alaa & Lawrence Jun Zhang
2024. Utility of Kolmogorov complexity measures: Analysis of L2 groups and L1 backgrounds. PLOS ONE 19:4 ► pp. e0301806 ff. DOI logo
Kaatari, Henrik, Ying Wang & Tove Larsson
2024. Introducing the Swedish Learner English Corpus: a corpus that enables investigations of the impact of extramural activities on L2 writing. Corpora 19:1 ► pp. 17 ff. DOI logo
Pun, Jack & Wangyin Kenneth Li
2024. A structural equation investigation of linguistic features as indices of writing quality in assessed secondary-level EMI learners’ scientific reports. Assessing Writing 62 ► pp. 100897 ff. DOI logo
Zheng, Yan & Jessie S. Barrot
2024. Syntactic complexity in second language (L2) writing: Comparing students’ narrative and argumentative essays. System 123 ► pp. 103342 ff. DOI logo
王, 泽文
2024. A Comparative Study of Phrasal Complexity in English Abstract Writing by Chinese and English-Native Writers. Modern Linguistics 12:11 ► pp. 584 ff. DOI logo
Hwang, Hyun-Bin & Charlene Polio
2023. Text length effects on the reliability of syntactic complexity indices. Research Methods in Applied Linguistics 2:3 ► pp. 100085 ff. DOI logo
Kaatari, Henrik, Tove Larsson, Ying Wang, Seda Acikara-Eickhoff & Pia Sundqvist
2023. Exploring the effects of target-language extramural activities on students’ written production. Journal of Second Language Writing 62 ► pp. 101062 ff. DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue