Cover not available

Article published In: International Journal of Corpus Linguistics
Vol. 24:1 (2019) ► pp.33–66

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Vocabulary sophistication in First-Year Composition assignments

 | University of Exeter
 | University of South Florida
 | University of Exeter
Published online: 2 July 2019
DOI logo with link
https://doi.org/10.1075/ijcl.17052.dur
Abstract
Recently-developed tools which quickly and reliably quantify vocabulary use on a range of measures open up new possibilities for understanding the construct of vocabulary sophistication. To take this work forward, we need to understand how these different measures relate to each other and to human readers’ perceptions of texts. This study applied 356 quantitative measures of vocabulary use generated by an automated vocabulary analysis tool (Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49 (4), 757–786. Google Scholar logo with link to Google Scholar) to a large corpus of assignments written for First-Year Composition courses at a university in the United States. Results suggest that the majority of measures can be reduced to a much smaller set without substantial loss of information. However, distinctions need to be retained between measures based on content vs. function words and on different measures of collocational strength. Overall, correlations with grades are reliable but weak.
Article outline
  • 1.Introduction
  • 2.Quantitative measures of vocabulary sophistication
    • Word length
    • Word frequency
    • Phraseological sophistication
  • 3.Methods
    • 3.1Corpus
    • 3.2Analysis
      • Frequency measures
      • Word range measures
      • N-gram proportion, frequency and association measures
      • Measures of academic vocabulary and phraseology
  • 4.Findings
    • 4.1Analysis by category
      • 4.1.1Word frequency
      • 4.1.2Word range
      • 4.1.3N-gram measures
      • 4.1.4Academic language
    • 4.2Principal components analysis
  • 5.Discussion
  • 6.Conclusions
  • Notes
  • References
References (63)
References
Berman, R. A., & Nir, B. (2010). The lexicon in writing-speech-differentiation. Written Language and Literacy, 13 (2), 183–205. Google Scholar logo with link to Google Scholar
Berman, R. A., & Nir-Sagiv, B. (2007). Comparing narrative and expository text construction across adolescence: A developmental paradox. Discourse Processes, 43 (2), 79–120. Google Scholar logo with link to Google Scholar
Bestgen, Y. (2017). Beyond single-word measures: L2 writing assessment, lexical richness and formulaic competence. System, 69 1, 65–78. Google Scholar logo with link to Google Scholar
Bestgen, Y., & Granger, S. (2014). Quantifying the development of phraseological competence in L2 English writing: An automated approach. Journal of Second Language Writing, 26 1, 28–41. Google Scholar logo with link to Google Scholar
Biber, D. (1988). Variation Across Speech and Writing. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
BNC Consortium. (2007). British National Corpus, version 3 (BNC XML ed.). Retrieved from [URL] (Last accessed February 2019).
Brown, G. D. A. (1984). A frequency count of 190,000 words in the London-Lund corpus of English conversation. Behavior Research Methods, Instrumentation & Computers, 16 (6), 502–532. Google Scholar logo with link to Google Scholar
Brysbaert, M., & New, B. (2009). Moving beyond Kucera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41 (4), 977–990. Google Scholar logo with link to Google Scholar
Bulté, B., & Housen, A. (2014). Conceptualizing and measuring short-term changes in L2 writing complexity. Journal of Second Language Writing, 26 1, 42–65. Google Scholar logo with link to Google Scholar
Burnage, G. (1990). CELEX: A Guide for Users. Nijmegen: CELEX – Centre for Lexical Information.Google Scholar logo with link to Google Scholar
Coxhead, A. (2000). A new academic wordlist. TESOL Quarterly, 34 (2), 213–238. Google Scholar logo with link to Google Scholar
Crossley, S. A., Cai, Z., & McNamara, D. (2012). Syntagmatic, paradigmatic, and automatic n-gram approaches to assessing essay quality. In G. M. Youngblood & P. M. McCarthy (Eds.), Proceedings of the Twenty-Fifth International Florida Artificial Intelligence Research Society Conference (pp. 214–219). Palo-Alto, CA: The AAAI Press.Google Scholar logo with link to Google Scholar
Crossley, S. A., DeFore, C., Kyle, K., Dai, J., & McNamara, D. (2013). Paragraph specific n-gram approaches to automatically assessing essay quality. In S. K. D’Mello, R. A. Clavo & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (pp. 216–219). Heidelberg: Springer. Retrieved from [URL] (Last accessed February 2019)
Crossley, S. A., Salsbury, T., McNamara, D., & Jarvis, S. (2010). Predicting lexical proficiency in language learner texts using computational indices. Language Testing, 28 (4), 561–580. Google Scholar logo with link to Google Scholar
Crossley, S. A., Weston, J. L., Sullivan, S. T. M., & McNamara, D. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28 (3), 282–311. Google Scholar logo with link to Google Scholar
Cumming, A., Kantor, R., Baba, K., Erdosy, U., Eouanzoui, K., & James, M. (2005). Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing, 10 (1), 5–43. Google Scholar logo with link to Google Scholar
Daller, H., Turlik, J., & Weir, I. (2013). Vocabulary acquisition and the learning curve. In S. Jarvis & H. Daller (Eds.), Vocabulary Knowledge: Human Ratings and Automated Measures (pp. 185–215). Amsterdam/Philadelphia, PA: John Benjamins. Google Scholar logo with link to Google Scholar
Davies, M. (2008-). The Corpus of Contemporary American: 450 million words, 1990-present. Retrieved from [URL] (last accessed February 2019).
Durrant, P. (2014). Corpus frequency and second language learners’ knowledge of collocations. International Journal of Corpus Linguistics, 19 (4), 443–477. Google Scholar logo with link to Google Scholar
Durrant, P., & Brenchley, M. (in press). Corpus research on the development of children’s writing in L1 English. In A. Glaznieks, A. Abel, V. Lyding, & V. Nicolas (Eds.), Corpora and Language in Use: Proceedings of the Learner Corpus Research Conference, 2017. Louvain: Presses Universitaires de Louvain.Google Scholar logo with link to Google Scholar
Durrant, P., & Schmitt, N. (2009). To what extent do native and non-native writers make use of collocations? International Review of Applied Linguistics, 47 (2), 157–177. Google Scholar logo with link to Google Scholar
Garner, J., Crossley, S. A., & Kyle, K. (2018). Beginning and intermediate L2 writers’ use of N-grams: An association measures study. International Review of Applied Linguistics. Advance online publication. Google Scholar logo with link to Google Scholar
Golub, L. S., & Frederick, W. C. (1979). Linguistic Structures in the discourse of fourth and sixth graders. Madison, WI: Center for Cognitive Learning, The University of Wisconsin.Google Scholar logo with link to Google Scholar
Graesser, A. C., McNamara, D., Louwerse, M. M., & Cai, Z. (2014). Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers, 36 (2), 193–202. Google Scholar logo with link to Google Scholar
Granger, S., & Bestgen, Y. (2014). The use of collocations by intermediate vs. advanced non-ntive writers: A bigram-based study. International Review of Applied Linguistics, 52 (3), 229–252. Google Scholar logo with link to Google Scholar
Gries, S. Th. (2013). 50-something years of work on collocations: What is or should be next… International Journal of Corpus Linguistics, 18 (1), 137–165. Google Scholar logo with link to Google Scholar
Grobe, C. (1981). Syntactic maturity, mechanics, and vocabulary as predictors of quality ratings. Research in the Teaching of English, 15 (1), 75–85.Google Scholar logo with link to Google Scholar
Guo, L., Crossley, S. A., & McNamara, D. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18 (3), 218–238. Google Scholar logo with link to Google Scholar
Hou, J., Verspoor, M., & Loerts, H. (2016). An exploratory study into the dynamics of Chinese L2 writing development. Dutch Journal of Applied Linguistics, 5 (1), 65–96. Google Scholar logo with link to Google Scholar
Jarvis, S., Grant, L., Bikowski, D., & Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12 1, 377–403. Google Scholar logo with link to Google Scholar
Kim, J.-Y. (2014). Predicting L2 writing proficiency using linguistic complexity measures: A corpus-based study. English Teaching, 69 (4), 27–51. Google Scholar logo with link to Google Scholar
Kim, M., Crossley, S. A., & Kyle, K. (2018). Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. The Modern Language Journal, 102 (1), 120–141. Google Scholar logo with link to Google Scholar
Knoch, U., Rouhshad, A., Oon, S. P., & Storch, N. (2015). What happens to ESL students’ writing after three years of study at an English medium university? Journal of Second Language Writing, 28 1, 39–52. Google Scholar logo with link to Google Scholar
Knoch, U., Rouhshad, A., & Storch, N. (2014). Does the writing of undergraduate ESL students develop after one year of study in an English-medium university? Assessing Writing, 21 1, 1–17. Google Scholar logo with link to Google Scholar
Kucera, H. & Francis, W. (1967). Computational Analysis of Present-day American English. Providence, RI: Brown University Press.Google Scholar logo with link to Google Scholar
Kyle, K. (2017). Modelling quality in source-based texts. Retrieved from [URL] (last accessed February 2019).
Kyle, K., & Crossley, S. A. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Quarterly, 49 (4), 757–786. Google Scholar logo with link to Google Scholar
(2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34 1, 12–24. Google Scholar logo with link to Google Scholar
Malvern, D., & Richards, B. (2002). Investigating accommodation in language proficiency interviews using a new measure of lexical diversity. Language Testing, 19 (1), 85–104. Google Scholar logo with link to Google Scholar
Malvern, D., Richards, B. J., Chipere, N., & Durán, P. (2004). Lexical Diversity and Language Development. Basingstoke: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Massey, A. J., & Elliott, G. L. (1996). Aspects of Writing in 16+ English Examinations Between 1980 & 1994. Cambridge: University of Cambridge Local Examinations Syndicate.Google Scholar logo with link to Google Scholar
Massey, A. J., Elliott, G. L., & Johnson, N. K. (2005). Variations in Aspects of Writing in 16+ English Examinations Between 1980 and 2004: Vocabulary, Spelling, Punctuation, Sentence Structure, Non-standard English. Cambridge: Cambridge Assessment.Google Scholar logo with link to Google Scholar
Mazgutova, D., & Kormos, J. (2015). Syntactic and lexical development in an intensive English for Academic Purposes programme. Journal of Second Language Writing, 29 1, 3–15. Google Scholar logo with link to Google Scholar
McCarthy, P. M., & Jarvis, S. (2011). MTLD, voc-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42 (2), 381–392. Google Scholar logo with link to Google Scholar
Meurers, D., & Dickinson, M. (2017). Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67 :S1, 66–95. Google Scholar logo with link to Google Scholar
Moxley, J. (2013). Big data, learning analytics, and social assessment. Journal of Writing Assessment, 6 (1), 1–10.Google Scholar logo with link to Google Scholar
Myhill, D. (1999). Writing matters: Linguistic characteristics of writing in GCSE English examinations. English in Education, 33 (3), 70–81. Google Scholar logo with link to Google Scholar
(2009). From talking to writing: Linguistic development in writing. BJEP Monograph Series II, 6 1, 27–44.Google Scholar logo with link to Google Scholar
Olinghouse, N. G., & Leaird, J. T. (2009). The relationship between measures of vocabulary and narrarive writing quality in second- and fourth-grade students. Reading and Writing, 22 1, 545–565. Google Scholar logo with link to Google Scholar
Olinghouse, N. G., & Wilson, J. (2013). The relationship between vocabulary and writing quality in three genres. Reading and Writing: An Interdisciplinary Journal, 26 1, 45–65. Google Scholar logo with link to Google Scholar
Paquot, M. (2018). Phraseological competence: A missing component in university entrance language tests? Insights from a study of EFL learners’ use of statistical collocations. Language Assessment Quarterly, 15 (1), 29–43. Google Scholar logo with link to Google Scholar
(2019). The phraseological dimension in interlanguage complexity research. Second Language Research, 35 (1), 121–145. Google Scholar logo with link to Google Scholar
R Development Core Team. (2013). R: A Language and Environment for Statistical Computing (Version 1.0.136) [Computer software]. Vienna: R Foundation for Statistical Computing. Retrieved from [URL] (last accessed February 2019).
Read, J. (2000). Assessing Vocabulary. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Roessingh, H., Elgie, S., & Kover, P. (2015). Using lexical profiling tools to investigage children’s written vocabulary in grade 3: An exploratory study. Language Assessment Quarterly, 12 (1), 67–86. Google Scholar logo with link to Google Scholar
Simpson-Vlach, R., & Ellis, N. C. (2010). An Academic Formulas List: New methods in phraseology research. Applied Linguistics, 31 (4), 487–512. Google Scholar logo with link to Google Scholar
Storch, N. (2009). The impact of studying in a second language (L2) medium university on the development of L2 writing. Journal of Second Language Writing, 18 (2), 103–118. Google Scholar logo with link to Google Scholar
Thorndike, E. L. & Lorge, I. (1944). The Teacher’s Word Book of 30,000 Words. New York, NY: Teachers College, Columbia University.Google Scholar logo with link to Google Scholar
Treffers-Daller, J., Parslow, P., & Williams, S. (2018). Back to basics: How measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics, 39 (3), 302–327.Google Scholar logo with link to Google Scholar
Uccelli, P., Dobbs, C. L., & Scott, J. (2013). Mastering academic language: Organization and stance in the persuasive writing of high school students. Written Communication, 30 (1), 36–62. Google Scholar logo with link to Google Scholar
Verspoor, M., Schmid, M. S., & Xu, X. (2012). A dynamic usage based perspective on L2 writing. Journal of Second Language Writing, 21 (3), 239–263. Google Scholar logo with link to Google Scholar
Vidakovic, I., & Barker, F. (2010). Use of words and multi-word units in Skills for Life Writing examinations. University of Cambridge ESOL Examinations Research Notes, 41 1, 7–14.Google Scholar logo with link to Google Scholar
Vieregge, Q., Stedman, K., Mitchell, T., & Moxley, J. (2012). Agency in the Age of Peer Production. Urbana, IL: National Council of Teachers of English.Google Scholar logo with link to Google Scholar
Cited by (13)

Cited by 13 other publications

Bottini, Raffaella & Elen Le Foll
2025. The more proficient the learners, the less sophisticated their L2 vocabulary?. International Journal of Learner Corpus Research 11:1 ► pp. 47 ff. DOI logo
Koubar, Souhair Hassan
2025. A remodeled cognitive vocabulary approach to assessing writing achievement in EFL education. The Journal of Educational Research 118:4 ► pp. 342 ff. DOI logo
Nicklin, Christopher, Daniel Bailey, Stuart McLean, Young Ae Kim, Hyeonah Kang & Joseph P. Vitta
2025. Applying lexical sophistication models to wordlist development: A proof-of-concept study. Research Methods in Applied Linguistics 4:1 ► pp. 100175 ff. DOI logo
Ruegg, Rachael
2025. Assessment of written assignments in first-year humanities and social sciences courses: textual features of academic writing. Assessment in Education: Principles, Policy & Practice 32:1 ► pp. 60 ff. DOI logo
Yoo, Hyunbin & Hyunwoo Kim
2025. A cross-modal analysis of lexical sophistication: EFL and ESL learners in written and spoken production. International Review of Applied Linguistics in Language Teaching 63:3 ► pp. 1795 ff. DOI logo
Friedman, Alon & Zachariah Beasley
2024. Using Textual Analysis to Examine Student Engagement in Online Undergraduate Science Education. Journal of Statistics and Data Science Education ► pp. 1 ff. DOI logo
Qian, Leyi
2023. Use of lexical features in high-stakes tests: Evidence from the perspectives of complexity, accuracy and fluency. Assessing Writing 57 ► pp. 100758 ff. DOI logo
Vitta, Joseph P., Christopher Nicklin & Simon W. Albright
2023. Academic word difficulty and multidimensional lexical sophistication: An English‐for‐academic‐purposes‐focused conceptual replication of Hashimoto and Egbert (2019). The Modern Language Journal 107:1 ► pp. 373 ff. DOI logo
McCallum, Lee & Philip Durrant
2022. Shaping Writing Grades, DOI logo
Stewart, Jeffrey, Joseph P. Vitta, Christopher Nicklin, Stuart McLean, Geoffrey G. Pinchbeck & Brandon Kramer
2022. The Relationship between Word Difficulty and Frequency: A Response to Hashimoto (2021). Language Assessment Quarterly 19:1 ► pp. 90 ff. DOI logo
Kim, Minkyung
2021. Exploring longitudinal changes in lexical and syntactic features in beginning-level EFL learner writing. System 103 ► pp. 102680 ff. DOI logo
Maamuujav, Undarmaa
2021. Examining lexical features and academic vocabulary use in adolescent L2 students’ text-based analytical essays. Assessing Writing 49 ► pp. 100540 ff. DOI logo
[no author supplied]
2022. Research Methods in Vocabulary Studies [Research Methods in Applied Linguistics, 2], DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue