Cover not available

Article published In: International Journal of Learner Corpus Research
Vol. 11:2 (2025) ► pp.245–275

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

The effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes exams

ORCiD logo with link | Luleå University of Technology
Published online: 13 January 2025
DOI logo with link
https://doi.org/10.1075/ijlcr.23038.hol
Abstract
The present study concerns the effect of lexical complexity on grading of Swedish EFL learners’ texts during high-stakes exams. A learner corpus consisting of 142 texts graded by expert raters and 175 texts graded by teachers was analysed to establish if the latter graded in agreement with the former as intended by the Swedish National Agency for Education (SNAE). Four indices of lexical complexity available in TAALED and TAALES were chosen to explore if this is the case. The method includes conducting ordinal regression with interactions to determine the effect of the independent variables on grade and if these variables have the same effect in texts graded by teachers and expert raters. The findings reveal a discrepancy between expert raters and teachers as they appear to consider lexical complexity to a different extent. It was also found that expert raters and teachers graded more in agreement during source-based writing tasks compared to independent writing tasks.
Article outline
  • 1.Introduction and aim
  • 2.Background
    • 2.1Lexical complexity and assessment of writing proficiency
    • 2.2Previous research on the assessment of Swedish EFL students
    • 2.3English as a subject in Swedish upper secondary school and the national tests
  • 3.Material and method
    • 3.1Material
    • 3.2Method
      • 3.2.1Lexical sophistication indices
      • 3.2.2Lexical diversity index
    • 3.3Statistical analysis
  • 4.Results
    • 4.1English 5
      • 4.1.1The effect of lexical complexity on grading
    • 4.2English 6
      • 4.2.1The effect of lexical complexity on grading
  • 5.Discussion
  • 6.Conclusion
  • Data availability statement
  • Notes
  • References
References (50)
References
Bergström, D., Norberg, C., & Nordlund, M. (2021). “Words are picked up along the way”– Swedish EFL teachers’ conceptualizations of vocabulary knowledge and learning. Language Awareness, 311, 393–409. Google Scholar logo with link to Google Scholar
Bottini, R., & Le Foll, E. (in press). The more proficient the learners, the less sophisticated their L2 vocabulary? The curious effect of the reference corpus on mean-frequency measures of lexical sophistication in written and spoken production. International Journal of Learner Corpus Research. Preprint available on: [URL]
Brysbaert, M., New, B., & Keuleers, E. (2012). Adding part-of-speech information to the SUBTLEX-US word frequencies. Behavior Research Methods, 441, 991–997. Google Scholar logo with link to Google Scholar
Christensen, R. (2023). ordinal — Regression Models for Ordinal Data. R package version, 2023.12-4.1. [URL]
Covington, M. A., & McFall, J. D. (2010). Cutting the gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94–100. Google Scholar logo with link to Google Scholar
Crossley, S. A., Roscoe, R., & McNamara, D. S. (2014). What is successful writing? An investigation into the multiple ways writers can write successful essays. Written Communication, 31(2), 184–214. Google Scholar logo with link to Google Scholar
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press. [URL]
Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10(1), 5–43. Google Scholar logo with link to Google Scholar
Educational Testing Service. (2023–05–30). Writing Scoring Guide. TOEFL iBT® Writing Scoring Guide Flyer (ets.org)Google Scholar logo with link to Google Scholar
Egbert, J., Larsson, T., & Biber, D. (2020). Doing linguistics with a corpus: Methodological considerations for the everyday user. Cambridge University Press. Google Scholar logo with link to Google Scholar
Egbert, J., Burch, B., & Biber, D. (2020). Lexical dispersion and corpus design. International Journal of Corpus Linguistics, 25(1), 89–115. Google Scholar logo with link to Google Scholar
Egbert, J., & Burch, B. (2023). Which words matter most? Operationalizing lexical prevalence for rank-ordered word lists. Applied Linguistics, 44(1), 103–126. Google Scholar logo with link to Google Scholar
Eguchi, M., & Kyle, K. (2020). Continuing to explore the multidimensional nature of lexical sophistication: The case of oral proficiency interviews. The Modern Language Journal, 104(2), 381–400. Google Scholar logo with link to Google Scholar
Engber, C. A. (1995). The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing, 4(2), 139–155. Google Scholar logo with link to Google Scholar
Erickson, G., Kao, T., & Lin, Y. (2010). Good practice in language testing and assessment: A matter of responsibility and respect. A new look at teaching and testing: English as subject and vehicle. Selected papers from the 2009 LTTC International Conference on English Language Teaching and Testing (pp. 237–258). University of Gothenburg.Google Scholar logo with link to Google Scholar
Erickson, G. (2020). National assessment of foreign languages in Sweden. University of Gothenburg. [URL]
Erickson, G., & Tholin, J. (2022). Overall, a good test, but… — Swedish lower secondary teachers’ perceptions and use of national test results of English. Languages, 7(1), 64. Google Scholar logo with link to Google Scholar
Fox, J., Weisberg, S., Price, B., Adler, D., Bates, D., Baud-Bovy, G., Bolker, B., Ellison, S., Firth, D., Friendly, M., Gorjanc, G., Graves, S., Heidelberger, R., Krivitsky, P., Laboissiere, R., Maechler, M., Monette, G., Murdoch, D., Nilsson, H., … & R-Core. (2012). Package ‘car’ version, 2024 3.1–3. Vienna: R Foundation for Statistical Computing, 16(332), 333. [URL]
Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. Google Scholar logo with link to Google Scholar
Hashimoto, B. J., & Egbert, J. (2019). More than frequency? Exploring predictors of word difficulty for second language learners. Language Learning, 69(4), 839–872. Google Scholar logo with link to Google Scholar
IELTS. (2023–05–30). Writing Band Descriptors. Writing Band Descriptors (ielts.org)Google Scholar logo with link to Google Scholar
Jarvis, S. (2002). Short texts, best-fitting curves and new measures of lexical diversity. Language Testing, 19(1), 57–84. Google Scholar logo with link to Google Scholar
Kaatari, H., Wang, Y., & Larsson, T. (2024). Introducing the Swedish Learner English Corpus: A corpus that enables investigations of the impact of extramural activities on L2 writing. Corpora, 19(1), 17–30. Google Scholar logo with link to Google Scholar
Kim, M., Crossley, S. A., & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102(1), 120–141. Google Scholar logo with link to Google Scholar
Kleiber, C., Zeileis, A., & Zeileis, M. A. (2020). R package ‘aer’ version, 1.2–14. AER: Applied Econometrics with R (r-project.org)Google Scholar logo with link to Google Scholar
Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods, 441, 978–990. Google Scholar logo with link to Google Scholar
Kyle, K. (2022). Measuring lexical richness. In S. Webb (Ed.), The Routledge Handbook of Vocabulary Studies (pp. 454–476). Routledge.Google Scholar logo with link to Google Scholar
Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49(4), 757–786. Google Scholar logo with link to Google Scholar
Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 341, 12–24. Google Scholar logo with link to Google Scholar
Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): version 2.0. Behavior Research Methods, 501, 1030–1046. Google Scholar logo with link to Google Scholar
Kyle, K., Crossley, S. A., & Jarvis, S. (2021). Assessing the validity of lexical diversity indices using direct judgements. Language Assessment Quarterly, 18(2), 154–170. Google Scholar logo with link to Google Scholar
Laufer, B., & Nation, P. (1995). Vocabulary size and use: Lexical richness in L2 written production. Applied Linguistics, 16(3), 307–322. Google Scholar logo with link to Google Scholar
Mangiafico, S. (2024). rcompanion: Functions to support extension education program evaluation. Rutgers cooperative xxtension, New Brunswick, New Jersey. version 2.4.36, [URL]
Plakans, L., & Gebril, A. (2013). Using multiple texts in an integrated writing assessment: Source text use as a predictor of score. Journal of Second Language Writing, 22(3), 217–230. Google Scholar logo with link to Google Scholar
R Development Core Team. (2023). R: A language and environment for statistical computing. [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. [URL]
Read, J. (2000). Assessing Vocabulary. Cambridge University Press. Google Scholar logo with link to Google Scholar
Schlegel, B. E., Steenbergen, M., Schlegel, M. B., & Imports, M. A. S. S. (2018). R Package brant, version 0.3–0. brant: Test for Parallel Regression Assumption (r-project.org)Google Scholar logo with link to Google Scholar
Schoonen, R., Van Gelderen, A., Stoel, R. D., Hulstijn, J., & De Glopper, K. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61(1), 31–79. Google Scholar logo with link to Google Scholar
Skolverket. (2022). Ämnesplan i Engelska. Skolverket. [URL]
University of Gothenburg. (2023–05–30a). Översikt — Skriftlig produktion och interaktion, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)Google Scholar logo with link to Google Scholar
. (2023–05–30b). Exempel på texter bedömda som E, Engelska 5. Exempel på uppgiftstyper för Engelska 5 | Projektet Nationella prov i främmande språk — Nafs, Göteborgs universitet (gu.se)Google Scholar logo with link to Google Scholar
Venables, W. N., & Ripley, B. D. (2002). Modern Applied Statistics with S. (4th ed.) Springer. Google Scholar logo with link to Google Scholar
Vögelin, C., Jansen, T., Keller, S. D., Machts, N., & Möller, J. (2019). The influence of lexical features on teacher judgements of ESL argumentative essays. Assessing Writing, 391, 50–63. Google Scholar logo with link to Google Scholar
Yang, W. (2014). Mapping the relationships among the cognitive complexity of independent writing tasks, L2 writing quality, and complexity, accuracy and fluency of L2 writing [Doctoral dissertation, Georgia State University]. ScholarWorks@Georgia State University.
Yang, W., Lu, X., & Weigle, S. C. (2015). Different topics, different discourse: Relationships among writing topic, measures of syntactic complexity, and judgments of writing quality. Journal of Second Language Writing, 281, 53–67. Google Scholar logo with link to Google Scholar
Yang, W., & Kim, Y. (2020). The effect of topic familiarity on the complexity, accuracy, and fluency of second language writing. Applied Linguistics Review, 11(1), 79–108. Google Scholar logo with link to Google Scholar
Yoon, H. J. (2017). Linguistic complexity in L2 writing revisited: Issues of topic, proficiency, and construct multidimensionality. System, 661, 130–141. Google Scholar logo with link to Google Scholar
Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236–259. Google Scholar logo with link to Google Scholar
Zenker, F., & Kyle, K. (2021). Investigating minimum text lengths for lexical diversity indices. Assessing Writing, 471, 100505. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue