Cover not available

Article published In: ITL - International Journal of Applied Linguistics
Vol. 170:1 (2019) ► pp.24–52

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Lexical aspects of comprehensibility and nativeness from the perspective of native-speaking English raters

 | Waseda University
 | Concordia University
 | Birkbeck, University of London
 | University College London
 | Western University
Published online: 5 April 2019
DOI logo with link
https://doi.org/10.1075/itl.17026.app
Abstract
This study analyzed the contribution of lexical factors to native-speaking raters’ assessments of comprehensibility and nativeness in second language (L2) speech. Using transcribed samples to reduce non-lexical sources of bias, 10 naïve L1 English raters evaluated speech samples from 97 L2 English learners across two tasks (picture description and TOEFL integrated). Subsequently, the 194 transcripts were analyzed through statistical software (e.g., Coh-metrix, VocabProfile) for 29 variables spanning various lexical dimensions. For the picture description task, separation in lexical correlates of the two constructs was found, with distinct lexical measures tied to comprehensibility and nativeness. In the TOEFL integrated task, comprehensibility and nativeness were largely indistinguishable, with identical sets of lexical variables, covering dimensions of diversity and range. Findings are discussed in relation to the acquisition, assessment, and teaching of lexical properties in L2 speech.
Article outline
  • Research on L2 speaking
  • L2 comprehensibility versus nativeness
  • The current study
  • Method
    • Speakers
    • Speaking tasks
    • Materials
    • Raters
    • Rating procedure
  • Lexical analysis
  • Coh-metrix measures
  • VocabProfile measures
  • Results
    • Comprehensibility and nativeness ratings
    • Lexical variables and rated constructs
    • Lexical predictors of comprehensibility and nativeness
  • Discussion
    • Lexical correlates of speech ratings
    • Comprehensibility versus nativeness
    • Task effects
  • Implications
  • Limitations and future research
  • Conclusion
  • Notes
  • References
References (55)
References
Abercrombie, D. (1949). Teaching pronunciation. ELT Journal, 31, 113–122. Google Scholar logo with link to Google Scholar
Adams, M. L. (1980). Five co-occurring factors in speaking proficiency. In J. R. Firth (Ed.), Measuring spoken language proficiency (pp. 1–6). Washington, DC: Georgetown University Press.Google Scholar logo with link to Google Scholar
Anderson, R. C., & Freebody, P. (1981). Vocabulary knowledge. In J. T. Gutherie (Ed.) Comprehension and teaching: Researching reviews (pp. 77–117). Newark, DE. International Reading Association.Google Scholar logo with link to Google Scholar
Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). CELEX. Philadelphia, PA: Linguistic Data Consortium.Google Scholar logo with link to Google Scholar
Browne, C., Culligan, B., & Phillips, J. (2013). The new general service list. Retrieved from <[URL]>
Cobb, T. (2016). Compleat Lexical Tutor [computer program]. <[URL]> (15 January 2016).
Crossley, S. A., Cobb, T., & McNamara, D. (2013). Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System, 411, 965–981. Google Scholar logo with link to Google Scholar
Crossley, S. A., & McNamara, D. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 171, 171–192.Google Scholar logo with link to Google Scholar
Crossley, S. A., Salsbury, T., & McNamara, D. S. (2009). Measuring L2 lexical growth using hypernymic relationships. Language Learning, 591, 307–334. Google Scholar logo with link to Google Scholar
(2010). The development of polysemy and frequency use in English second language speakers. Language Learning, 601, 573–605. Google Scholar logo with link to Google Scholar
(2015). Assessing lexical proficiency using analytic ratings: A case for collocation accuracy. Applied Linguistics, 361, 570–590.Google Scholar logo with link to Google Scholar
Crossley, S. A., Salsbury, T., McNamara, D. S., & Jarvis, S. (2011). What is lexical proficiency? Some answers from computational models of speech data. TESOL Quarterly, 451, 182–193. Google Scholar logo with link to Google Scholar
Crossley, S. A., Yang, H. S., & McNamara, D. S. (2014). What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 261, 92–113.Google Scholar logo with link to Google Scholar
Crowther, D., Trofimovich, P., Isaacs, T., & Saito, K. (2015). Does a speaking task affect second language comprehensibility? The Modern Language Journal, 991, 80–95. Google Scholar logo with link to Google Scholar
(2013). The development of L2 oral language skills in two L1 groups: A seven-year study. Language Learning, 631, 163–185. Google Scholar logo with link to Google Scholar
Derwing, T. M., Rossiter, M. J., & Ehrensberger-Dow, M. (2002). They speaked and wrote real good: Judgements of non-native and native grammar. Language Awareness, 111, 84–99. Google Scholar logo with link to Google Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). L2 fluency: Judgements on different tasks. Language Learning, 541, 655–679. Google Scholar logo with link to Google Scholar
Educational Testing Services (2004). Independent Speaking Scoring Rubrics. Retrieved from <[URL]>
Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.Google Scholar logo with link to Google Scholar
Ginther, A., Dimova, S., & Yang, R. (2010). Conceptual and empirical relationships between temporal measures of fluency and oral English proficiency with implications for automated scoring. Language Testing, 271, 379–399. Google Scholar logo with link to Google Scholar
Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-Metrix providing multilevel analyses of text characteristics. Educational Researcher, 401, 223–234. Google Scholar logo with link to Google Scholar
Granena, G. (2014). Individual differences in sequence learning ability and second language acquisition in early childhood and adulthood. Language Learning, 631, 665–703. Google Scholar logo with link to Google Scholar
Iwashita, N., Brown, A., McNamara, T., & O’Hagan, S. (2008). Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics, 291, 24–49. Google Scholar logo with link to Google Scholar
Kang, O., Rubin, D., Pickering, L. (2010). Suprasegmental measures of accentedness and judgements of English language learner proficiency in oral English. The Modern Language Journal, 941, 554–566. Google Scholar logo with link to Google Scholar
Koizumi, R., & In’nami, Y. (2012). Effects of text length on lexical diversity measures: Using short texts with less than 200 tokens. System, 401, 554–564. Google Scholar logo with link to Google Scholar
Kormos, J., & Dénes, M. (2004). Exploring measures and perceptions of fluency in the speech of second language learners. System, 321, 145–164. Google Scholar logo with link to Google Scholar
Kuiken, F., & Vedder, I. (2014). Raters’ decisions, rating procedures and rating scales. Language Testing, 311, 279–284. Google Scholar logo with link to Google Scholar
Kyle, K., & Crossley, S. (2015). Automatically assessing lexical sophistication: Indices, tools, findings, and applications. TESOL Quarterly, 491, 757–786. Google Scholar logo with link to Google Scholar
Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.Google Scholar logo with link to Google Scholar
Levis, J. (2005). Changing contexts and shifting paradigms in pronunciation teaching. TESOL Quarterly, 391, 369–377. Google Scholar logo with link to Google Scholar
Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Review, 961, 190–208. Google Scholar logo with link to Google Scholar
McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
McNamara, T. F. (1990). Item response theory and the validation of an ESP test for health professionals. Language Testing, 71, 52–75. Google Scholar logo with link to Google Scholar
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., & Miller, K. J. (1990). Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 31, 235–244. Google Scholar logo with link to Google Scholar
Moyer, A. (2013). Foreign accent: The phenomenon of non-native speech. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 341, 520–531. Google Scholar logo with link to Google Scholar
(2009). Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. The Canadian Modern Language Review, 661, 181–202. Google Scholar logo with link to Google Scholar
Munro, M. J., & Derwing, T. M., Burgess, C. S. (2010). Detection of nonnative speaker status from content-masked speech. Speech Communication, 521, 626–637. Google Scholar logo with link to Google Scholar
Nation, I. S. P. (2012). The BNC/COCA word family lists (17 September 2012). Unpublished paper. Available at <[URL]>
Neufeld, S., & Billuroğlu, A. (2005). In search of the critical lexical mass: How ‘general’ is the GSL? How ‘academic’ is the AWL? Available at <[URL]>
Oppenheimer, D. M. (2008). The secret life of fluency. Trends in Cognitive Sciences, 121, 237–241. Google Scholar logo with link to Google Scholar
Perfetti, C. A., Landi, N., & Oakhill, J. The acquisition of reading comprehension skill. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 227–247). Oxford: Blackwell.
Plonsky, L., & Oswald, F. L. (2014). How big is “big”? Interpreting effects sizes in L2 research. Language Learning, 641, 878–912. Google Scholar logo with link to Google Scholar
Read, J. (2004). Plumbing the depths. In P. Bogaards & B. Laufer (Eds.), Vocabulary in second language (pp. 209–227). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Révész, A., Michel, M., & Gilabert, R. (2015). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgements. Studies in Second Language Acquisition, 38(4), 703–737. Google Scholar logo with link to Google Scholar
Robinson, P. (2005). Cognitive complexity and task sequencing: Studies in a componential framework for second language task design. International Review of Applied Linguistics in Language Teaching, 431, 1–32. Google Scholar logo with link to Google Scholar
Saito, K., Trofimovich, P., Isaacs, T. (2016). Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics, 371, 217–240. Google Scholar logo with link to Google Scholar
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (2015). Lexical profiles of comprehensible second language speech. Studies in Second Language Acquisition, 381, 677–701. Google Scholar logo with link to Google Scholar
Salsbury, T., Crossley, S. A., & McNamara, D. S. (2011). Psycholinguistic word information in second language oral discourse. Second Language Research, 271, 343–360. Google Scholar logo with link to Google Scholar
Trofimovich, P., & Isaacs, T. (2012). Disentangling accent from comprehensibility. Bilingualism: Language and Cognition, 151, 905–916. Google Scholar logo with link to Google Scholar
Wen, Q., Wang, L., & Liang, M. (2005). Spoken and written English corpus of Chinese learners. Beijing: Foreign Language Teaching and Research Press.Google Scholar logo with link to Google Scholar
Wilson, M. D. (1988). The MRC psycholinguistic database: Machine readable dictionary. Behavioural Research Methods, Instruments and Computers, 201, 6–11. Google Scholar logo with link to Google Scholar
Wray, A. (2002). Formulaic language and the lexicon. New York, NY: Cambridge University Press. Google Scholar logo with link to Google Scholar
Cited by (12)

Cited by 12 other publications

Tergujeff-Vasu, Elina
2025. What Constitutes Second Language Comprehensibility and Accentedness? Evidence From New Speaker Groups. Journal of Monolingual and Bilingual Speech 6:3 ► pp. 289 ff. DOI logo
Díez-Ortega, María & Kristopher Kyle
2024. Measuring the development of lexical richness of L2 Spanish: A longitudinal learner corpus study. Studies in Second Language Acquisition 46:1 ► pp. 169 ff. DOI logo
Tsunemoto, Aki & Pavel Trofimovich
2024. Coherence and comprehensibility in second language speakers’ academic speaking performance. Studies in Second Language Acquisition 46:3 ► pp. 795 ff. DOI logo
Wang-Taylor, Yixin, Jon Clenton & Yinna Ren
2024. Investigating the Impact of Dialogic and Trialogic Interactive Factors on Chinese Advanced L2 learners’ Vocabulary Use in Spoken Contexts. Languages 9:8 ► pp. 266 ff. DOI logo
Appel, Randy & Angel Arias
2023. Automated Measures of Lexical Sophistication: Predicting Proficiency in an Integrated Academic Writing Task. Journal of Language and Education 9:1 ► pp. 12 ff. DOI logo
Shao, Yujie, Kazuya Saito & Adam Tierney
2023. How Does Having a Good Ear Promote Instructed Second Language Pronunciation Development? Roles of Domain‐General Auditory Processing in Choral Repetition Training. TESOL Quarterly 57:1 ► pp. 33 ff. DOI logo
Wang-Taylor, Yixin & Jon Clenton
2022. Exploring the relationship between L2 vocabulary size and academic speaking. System 107 ► pp. 102822 ff. DOI logo
Saito, Kazuya
2021. What Characterizes Comprehensible and Native‐like Pronunciation Among English‐as‐a‐Second‐Language Speakers? Meta‐Analyses of Phonological, Rater, and Instructional Factors. TESOL Quarterly 55:3 ► pp. 866 ff. DOI logo
Saito, Kazuya, Yui Suzukida, Mai Tran & Adam Tierney
2021. Domain‐General Auditory Processing Partially Explains Second Language Speech Learning in Classroom Settings: A Review and Generalization Study. Language Learning 71:3 ► pp. 669 ff. DOI logo
Suzukida, Yui & Kazuya Saito
2021. Which segmental features matter for successful L2 comprehensibility? Revisiting and generalizing the pedagogical value of the functional load principle. Language Teaching Research 25:3 ► pp. 431 ff. DOI logo
Noreillie, Ann-Sophie, Piet Desmet & Elke Peters
2020. Factors Predicting Low-Intermediate French Learners’ Vocabulary Use in Speaking Tasks. The Canadian Modern Language Review 76:3 ► pp. 194 ff. DOI logo
Saito, Kazuya, Mai Tran, Yui Suzukida, Hui Sun, Viktoria Magne & Meltem Ilkan
2019. HOW DO SECOND LANGUAGE LISTENERS PERCEIVE THE COMPREHENSIBILITY OF FOREIGN-ACCENTED SPEECH?. Studies in Second Language Acquisition 41:5 ► pp. 1133 ff. DOI logo

This list is based on CrossRef data as of 30 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue