Cover not available

Article published In: 25 years of Intelligibility, Comprehensibility and Accentedness
Edited by John M. Levis, Tracey M. Derwing and Murray J. Munro
[Journal of Second Language Pronunciation 6:3] 2020
► pp. 402–429

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Reactions to second language speech

Influences of discrete speech characteristics, rater experience, and speaker first language background

ORCiD logo with link | University College London
 | Brock University
Published online: 1 September 2020
DOI logo with link
https://doi.org/10.1075/jslp.20018.isa
Abstract
This study investigates how Mandarin and Slavic language speakers’ comprehensibility, accentedness, and fluency ratings, as assigned by experienced teacher-raters and novice raters, align with discrete linguistic measures, and raters’ accounts of influences on their scoring. In addition to examining mean ratings in relation to rater experience and speaker first language background, we correlated ratings with segmental, prosodic, and temporal measures. Introspective reports were segmented, coded, enumerated, and submitted to loglinear analysis to elucidate influences on ratings. Results showed that ratings were strongly correlated with prosodic goodness and moderately correlated with segmental errors, implying the importance of both segmentals and prosody in L2 speech ratings. Experienced teacher-raters provided lengthier reports than novice raters, producing more comments for all coded categories where an error was identified except for pausing (a dysfluency marker). This may be because novice raters observed little else about the speech or struggled to pinpoint or articulate other features.
Article outline
  • 1.Introduction
  • 2.The current study
  • 3.Method
    • 3.1Research design
    • 3.2L2 speakers
    • 3.3Speech elicitation and data preparation
    • 3.4Raters
    • 3.5Rating sessions
    • 3.6Rating scale normalization
    • 3.7Deriving discrete linguistic measures from the L2 speech samples
    • 3.8Analysis of introspective reports
  • 4.Results
    • 4.1Preliminary analyses
    • 4.2Rater experience and speaker L1 in relation to global ratings and discrete measures
    • 4.3Analysis of the factors that raters reportedly take notice of when rating L2 speech
  • 5.Discussion
    • 5.1Rater experience
    • 5.2Speaker L1 background
    • 5.3Concluding remarks
  • Acknowledgments
  • References
References (47)
References
Bannigan, K., & Watson, R. (2009). Reliability and validity in a nutshell. Journal of Clinical Nursing, 18(23), 3237–3243. Google Scholar logo with link to Google Scholar
Bongaerts, T., Mennen, S., & van der Slik, F. (2000). Authenticity of pronunciation in naturalistic second language acquisition: The case of very advanced late learners of Dutch as a second language. Studia Linguistica, 54(2), 298–308. Google Scholar logo with link to Google Scholar
Bongaerts, T., van Summeren, C., Planken, B., & Schils, E. (1997). Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 19(4), 447–465. Google Scholar logo with link to Google Scholar
Bradlow, A., Clopper, C., Smiljanic, R., & Walter, M. A. (2010). A perceptual phonetic similarity space for languages: Evidence from five native language listener groups. Speech Communication, 52(11–12), 930–942. Google Scholar logo with link to Google Scholar
Browne, K., & Fulcher, G. (2017). Pronunciation and intelligibility in assessing spoken fluency. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation: Interdisciplinary perspectives (pp. 37–53). Bristol, UK: Multilingual Matters.Google Scholar logo with link to Google Scholar
Chalhoub-Deville, M. (1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12(1), 62–70. Google Scholar logo with link to Google Scholar
Creswell, J. W., & Plano Clark, V. L. (2017). Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: SAGE.Google Scholar logo with link to Google Scholar
Crowther, D., Trofimovich, P., Saito, K., & Isaacs, T. (2015). Second language comprehensibility revisited: Investigating the effects of learner background. TESOL Quarterly, 49(4), 814–837. Google Scholar logo with link to Google Scholar
de Boer, M., & Heeren, W. (2019). The speaker-specificity of filled pauses: A cross-linguistic study. Proceedings of the International Congress of Phonetic Sciences (ICPhS) 2019 (pp. 607–611). Melbourne, Australia: Australasian Speech Science and Technology Association.Google Scholar logo with link to Google Scholar
Derwing, T. M., & Munro, M. J. (1997). Accent, intelligibility, and comprehensibility: Evidence from four L1s. Studies in Second Language Acquisition, 19(1), 1–16. Google Scholar logo with link to Google Scholar
(2009). Comprehensibility as a factor in listener interaction preferences: Implications for the workplace. Canadian Modern Language Review, 66(2), 181–202. Google Scholar logo with link to Google Scholar
(2013). The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 63(2), 163–185. Google Scholar logo with link to Google Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 54(4), 665–679. Google Scholar logo with link to Google Scholar
Derwing, T. M., Thomson, R. I., & Munro, M. J. (2006). English pronunciation and fluency development in Mandarin and Slavic speakers. System, 34(2), 183–193. Google Scholar logo with link to Google Scholar
Douglas, D. (1994). Quantity and quality in speaking test performance. Language Testing, 11(1), 125–144. Google Scholar logo with link to Google Scholar
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press. Google Scholar logo with link to Google Scholar
Foote, J. A., Isaacs, T., & Trofimovich, P. (2013, June 3–5). Developing a teacher-friendly assessment tool for L2 comprehensibility. Canadian Association of Applied Linguistics (ACLA/CAAL) conference, Calgary, AB.
Foote, J. A., & Trofimovich, P. (2018). Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 74(2), 253–278. Google Scholar logo with link to Google Scholar
Gass, S. M., & Mackey, A. (2000). Stimulated recall methodology in second language research. Mahwah, NJ: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Hahn, L. D. (2004). Primary stress and intelligibility: Research to motivate the teaching of suprasegmentals. TESOL Quarterly, 38(2), 201–233. Google Scholar logo with link to Google Scholar
Isaacs, T., & Harding, L. (2017). Research timeline: Pronunciation assessment. Language Teaching, 50(3), 347–366. Google Scholar logo with link to Google Scholar
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10(2), 135–159. Google Scholar logo with link to Google Scholar
Isaacs, T., & Trofimovich, P. (2012). Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 34(3), 475–505. Google Scholar logo with link to Google Scholar
Isaacs, T., Trofimovich, P., Yu, G., & Chereau, B. M. (2015). Examining the linguistic aspects of speech that most efficiently discriminate between upper levels of the revised IELTS pronunciation scale. IELTS research reports online series, 41.Google Scholar logo with link to Google Scholar
Kang, O., & Moran, M. (2014). Functional loads of pronunciation features in nonnative speakers’ oral assessment. TESOL Quarterly, 48(1), 176–187. Google Scholar logo with link to Google Scholar
Lennon, P. (1990). Investigating fluency in EFL: A quantitative approach. Language Learning, 40(3), 387–417. Google Scholar logo with link to Google Scholar
Lumley, T. (2005). Assessing second language writing: The rater’s perspective. Frankfurt: Peter Lang.Google Scholar logo with link to Google Scholar
McAndrews, M. M., & Thomson, R. I. (2017). Establishing an empirical basis for priorities in pronunciation teaching. Journal of Second Language Pronunciation, 3(2), 267–287. Google Scholar logo with link to Google Scholar
Munro, M. J. (2018). Dimensions of pronunciation. In O. Kang, R. Thomson, & J. Murphy. The Routledge handbook of contemporary English pronunciation (pp. 413–431). New York: Routledge.Google Scholar logo with link to Google Scholar
Munro, M. J., & Derwing, T. M. (2006). The functional load principle in ESL pronunciation instruction: An exploratory study. System, 34(4), 520–531. Google Scholar logo with link to Google Scholar
O’Brien, M. G. (2016). Methodological choices in rating speech samples. Studies in Second Language Acquisition, 38(3), 587–605. Google Scholar logo with link to Google Scholar
Pawlikowska-Smith, G. (2000). Canadian Language Benchmarks 2000: Theoretical framework. Ottawa, ON: Centre for Canadian Language Benchmarks.Google Scholar logo with link to Google Scholar
Rajadurai, J. (2007). Intelligibility studies: A consideration of empirical and ideological issues. World Englishes, 26(1), 87–98. Google Scholar logo with link to Google Scholar
Riggenbach, H. (1991). Toward an understanding of fluency: A microanalysis of non-native speaker conversations. Discourse Processes, 14(4), 423–441. Google Scholar logo with link to Google Scholar
Rose, H., & Galloway, N. (2019). Global Englishes for language teaching. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 65(3), 395–412. Google Scholar logo with link to Google Scholar
Saito, K., Trofimovich, P., & Isaacs, T. (2016). Second language speech production: Investigating linguistic correlates of comprehensibility and accentedness for learners at different ability levels. Applied Psycholinguistics, 37(2), 217–240. Google Scholar logo with link to Google Scholar
Saito, K., Trofimovich, P., Isaacs, T., & Webb, S. (2017). Re-examining phonological and lexical correlates of second language comprehensibility: The role of rater experience. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment: Interdisciplinary perspectives (pp. 131–146). Bristol, UK: Multilingual Matters.Google Scholar logo with link to Google Scholar
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (2016). Lexical correlates of comprehensibility versus accentedness in second language speech. Bilingualism: Language and Cognition, 19(3), 597–609. Google Scholar logo with link to Google Scholar
Schiavetti, N. (1992). Scaling procedures for the measurement of speech intelligibility. In R. D. Kent (Ed.), Intelligibility in speech disorders (pp. 11–34). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Taylor & Francis.Google Scholar logo with link to Google Scholar
Suzukida, Y., & Saito, K. (2019). Which segmental features matter for successful L2 comprehensibility? Revisiting and generalizing the pedagogical value of the Functional Load principle. Language Teaching Research. Advance online publication. Google Scholar logo with link to Google Scholar
Thomson, R. I., & Isaacs, T. (2009). Within-category variation in L2 English vowel learning. Canadian Acoustics, 371, 138–139.Google Scholar logo with link to Google Scholar
Thompson, I. (1991). Foreign accents revisited: The English pronunciation of Russian immigrants. Language Learning, 41(2), 177–204. Google Scholar logo with link to Google Scholar
Upshur, J. A., & Turner, C. E. (1999). Systematic effects in the rating of second-language speaking ability: Test method and learner discourse. Language Testing, 16(1), 82–111. Google Scholar logo with link to Google Scholar
Winke, P., Gass, S., & Myford, C. (2013). Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 30(2), 231–252. Google Scholar logo with link to Google Scholar
Zielinski, B. W. (2008). The listener: No longer the silent partner in reduced intelligibility. System, 36(1), 69–84. Google Scholar logo with link to Google Scholar
Cited by (10)

Cited by ten other publications

Chau, Tuc & Amanda Huensch
2025. The relationships among L2 fluency, intelligibility, comprehensibility, and accentedness. Studies in Second Language Acquisition 47:1 ► pp. 282 ff. DOI logo
Magne, Viktoria, Yui Suzukida & Walcir Cardoso
2025. Linguistic correlates of second language users’ attitudes to Arabic and Chinese varieties of English: a verbal guise study. Asian Englishes 27:2 ► pp. 287 ff. DOI logo
Borgia, Kathleen
2024. La compréhensibilité en français Lx : effort et contribution des locuteurs L1. La Revue de l’AQEFLS 36:1 DOI logo
Kermad, Alyssa
2024. TRAINING THE “EVERYDAY” LISTENER HOW TO RATE ACCENTED SPEECH. International Journal of Listening 38:1 ► pp. 58 ff. DOI logo
Levis, John M.
2024. Key issues in L2 pronunciation research. Journal of Second Language Pronunciation 10:3 ► pp. 293 ff. DOI logo
O'Grady, Stefan
2023. Halo effects in rating data: Assessing speech fluency. Research Methods in Applied Linguistics 2:2 ► pp. 100048 ff. DOI logo
Saito, Kazuya, Konstantinos Macmillan, Magdalena Kachlicka, Takuya Kunihara & Nobuaki Minematsu
2023. Automated assessment of second language comprehensibility: Review, training, validation, and generalization studies. Studies in Second Language Acquisition 45:1 ► pp. 234 ff. DOI logo
Lee, Andrew H. & Ron Thomson
2022. Pronunciation. In Research methods in instructed second language acquisition [Research Methods in Applied Linguistics, 3], ► pp. 233 ff. DOI logo
Thomson, Ron I. & Talia Isaacs
2022. Evaluations of Foreign Accented Speech: Subjective Bias or Speech Signal Characteristics?. In Theoretical and Practical Developments in English Speech Assessment, Research, and Training [Second Language Learning and Teaching, ], ► pp. 27 ff. DOI logo
SUZUKI, SHUNGO, JUDIT KORMOS & TAKUMI UCHIHARA
2021. The Relationship Between Utterance and Perceived Fluency: A Meta‐Analysis of Correlational Studies. The Modern Language Journal 105:2 ► pp. 435 ff. DOI logo

This list is based on CrossRef data as of 13 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue