Cover not available

Article published In: Journal of Second Language Pronunciation
Vol. 6:2 (2020) ► pp.236–264

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Testing the malleability of teachers’ judgments of second language speech

ORCiD logo with link | Concordia University, Canada
 | University of Calgary, Canada
 | Concordia University, Canada
 | University of Calgary, Canada
Published online: 9 March 2020
DOI logo with link
https://doi.org/10.1075/jslp.19015.tay
Abstract
This study examined whether a negative social bias can influence how teachers evaluate second language (L2) speech. Twenty-eight teachers of L2 German from Western Canada – 14 native speakers (NSs) and 14 proficient non-native speakers (NNSs) – rated recordings of 24 adult L2 learners of German across five speech dimensions (accentedness, comprehensibility, vowel/consonant accuracy, intonation, flow) using 1,000-point scales. Immediately before rating, half of NS and NNS teachers heard critical comments about undergraduate German students’ language skills, while the other half heard no biasing comments. Under negative bias, while the NNS teachers provided favorable evaluations across all five measures, NS teachers followed suit for only intonation and flow, downgrading L2 speakers’ accentedness, comprehensibility, and vowel/consonant accuracy. Findings call into question the relative stability of L2 speech ratings and highlight the importance of social context and teacher status as native versus non-native speakers of the target language in assessments of L2 speaking performance.
Article outline
  • 1.Introduction
  • 2.Background literature
    • 2.1Listener biases and their origins
    • 2.2Listener-specific factors
    • 2.3The current study
  • 3.Method
    • 3.1Teacher raters
    • 3.2Speech materials
    • 3.3Rating procedure
    • 3.4Data analysis
  • 4.Results
  • 5.Discussion
    • 5.1L2 speech ratings and listener bias
    • 5.2Sources of teacher bias
    • 5.3Relationship between teacher status and negative bias varies across speech dimensions
    • 5.4Uneven baselines in the absence of negative bias
  • 6.Implications and conclusion
  • Acknowledgements
  • Notes
  • References
References (72)
References
Anderson-Hsieh, J., Johnson, R., & Koehler, K. (1992). The relationship between native speaker judgments of nonnative pronunciation and deviance in segmentals, prosody, and syllable structure. Language Learning, 421, 529–555. Google Scholar logo with link to Google Scholar
Beukeboom, C. J., & Semin, G. R. (2006). How mood turns on language. Journal of Experimental Social Psychology, 421, 553–566. Google Scholar logo with link to Google Scholar
Bodenhausen, G. V., Mussweiler, T., Gabriel, S., & Moreno, K. N. (2001). Affective influences on stereotyping and intergroup relations. In J. P. Forgas (Ed.), Handbook of affect and social cognition (pp. 319–343). Mahwah, NJ: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Bongaerts, T., van Summeren, C., Planken, B., & Schils, E. (1997). Age and ultimate attainment in the pronunciation of a foreign language. Studies in Second Language Acquisition, 191, 447–465. Google Scholar logo with link to Google Scholar
Boughton, Z. (2006). When perception isn’t reality: Accent identification and perceptual dialectology in French. Journal of French Language Studies, 161, 277–304. Google Scholar logo with link to Google Scholar
Bradlow, A. R., & Bent, T. (2008). Perceptual adaptation to non-native speech. Cognition, 1061, 707–729. Google Scholar logo with link to Google Scholar
Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing, 121, 1–15. Google Scholar logo with link to Google Scholar
Bryk, A. S., & Schneider, B. (2003). Trust in schools: A core resource for school reform. Educational Leadership, 601, 40–45.Google Scholar logo with link to Google Scholar
Buckingham, L. (2014). Attitudes to English teachers’ accents in the Arabian Gulf. International Journal of Applied Linguistics, 241, 50–73. Google Scholar logo with link to Google Scholar
Carey, M. D., Mannell, R. H., & Dunn, P. K. (2011). Does a rater’s familiarity with a candidate’s pronunciation affect the rating in oral proficiency interviews? Language Testing, 281, 201–219. Google Scholar logo with link to Google Scholar
Cargile, A. C., & Giles, H. (1997). Understanding language attitudes: Exploring listener affect and identity. Language & Communication, 171, 195–217. Google Scholar logo with link to Google Scholar
Chiba, R., Matsuura, H., & Yamamoto, A. (1995). Japanese attitudes toward English accents. World Englishes, 141, 77–86. Google Scholar logo with link to Google Scholar
Crowther, D., Trofimovich, P., & Isaacs, T. (2016). Linguistic dimensions of second language accent and comprehensibility. Journal of Second Language Pronunciation, 21, 160–182. Google Scholar logo with link to Google Scholar
D’Onofrio, A. (2018). Controlled and automatic perceptions of a sociolinguistic marker. Language Variation and Change, 301, 261–285. Google Scholar logo with link to Google Scholar
(forthcoming). Sociolinguistic signs as cognitive representations. In L. Hall-Lew, E. Moore, & R. J. Podesva (Eds.), Social meaning and linguistic variation. Cambridge, UK: Cambridge University Press.
Dailey-O’Cain, J. (1999). The perception of post-unification German regional speech. In D. R. Preston (Ed.), Handbook of perceptual dialectology (Vol 11, pp. 227–242). Philadelphia, PA: John Benjamins. Google Scholar logo with link to Google Scholar
Dalton-Puffer, C., Kaltenböck, G., & Smit, U. (1997). Learner attitudes and L2 pronunciation in Austria. World Englishes, 161, 115–128. Google Scholar logo with link to Google Scholar
Derwing, T. M., & Munro, M. (2013). The development of L2 oral language skills in two L1 groups: A 7-year study. Language Learning, 631, 163–185. Google Scholar logo with link to Google Scholar
Derwing, T. M., Rossiter, M. J., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgments on different tasks. Language Learning, 541, 655–679. Google Scholar logo with link to Google Scholar
Eskenazi, M. (2009). An overview of spoken language technology for education. Speech Communication, 511, 832–844. Google Scholar logo with link to Google Scholar
Fayer, J. M., & Krasinski, E. (1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 371, 313–326. Google Scholar logo with link to Google Scholar
Fazio, R. H., Sanbonmatsu, D. M., Powell, M. C., & Kardes, F. R. (1986). On the automatic activation of attitudes. Journal of Personality and Social Psychology, 501, 229–238. Google Scholar logo with link to Google Scholar
Foote, J. A., & Trofimovich, P. (2018). Is it because of my language background? A study of language background influence on comprehensibility judgments. Canadian Modern Language Review, 741, 253–278. Google Scholar logo with link to Google Scholar
Ford, C. E. (1984). The influence of speech variety on teachers’ evaluation of students with comparable academic ability. TESOL Quarterly, 181, 25–40. Google Scholar logo with link to Google Scholar
Galinsky, A. D., & Moskowitz, G. B. (2000). Perspective-taking: Decreasing stereotype expression, stereotype accessibility, and in-group favoritism. Journal of Personality and Social Psychology, 781, 708–724. Google Scholar logo with link to Google Scholar
Goethe Institut (2004). Einstufungstest [Placement test]. [URL]
Hansen Edwards, J. G., Zampini, M. L., & Cunningham, C. (2018). The accentedness, comprehensibility, and intelligibility of Asian Englishes. World Englishes, 371, 538–557. Google Scholar logo with link to Google Scholar
(2019). Listener judgments of speaker and speech traits of varieties of Asian English. Journal of Multilingual and Multicultural Development, 401, 691–706. Google Scholar logo with link to Google Scholar
Hansen, K., Rakić, T., & Steffens, M. C. (2014). When actions speak louder than words: Preventing discrimination of nonstandard speakers. Journal of Language and Social Psychology, 331, 68–77. Google Scholar logo with link to Google Scholar
He, D., & Miller, L. (2011). English teacher preference: The case of China’s non-English-major students. World Englishes, 301, 428–443. Google Scholar logo with link to Google Scholar
Hu, G., & Lindemann, S. (2009). Stereotypes of Cantonese English, apparent native/non-native status, and their effect on non-native English speakers’ perception. Journal of Multilingual and Multicultural Development, 301, 253–269. Google Scholar logo with link to Google Scholar
Hu, G., & Su, J. (2015). The effect of native/non-native information on non-native listeners’ comprehension. Language Awareness, 241, 273–281. Google Scholar logo with link to Google Scholar
Hundt, M., Zipp, L., & Huber, A. (2015). Attitudes in Fiji towards varieties of English. World Englishes, 341, 688–707. Google Scholar logo with link to Google Scholar
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgments of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 101, 135–159. Google Scholar logo with link to Google Scholar
Kang, O. (2012). Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 91, 249–269. Google Scholar logo with link to Google Scholar
Kang, O., & Rubin, D. L. (2009). Reverse linguistic stereotyping: Measuring the effect of listener expectations on speech evaluation. Journal of Language and Social Psychology, 281, 441–456. Google Scholar logo with link to Google Scholar
Kawakami, K., Phills, C. E., Greenwald, A. G., Simard, D., Pontiero, J., Brnjas, A., & Dovidio, J. F. (2012). In perfect harmony: Synchronizing the self to activated social categories. Journal of Personality and Social Psychology, 1021, 562–575. Google Scholar logo with link to Google Scholar
Lambert, W. E., Hodgson, R. C., Gardner, R. C., & Fillenbaum, S. (1960). Evaluational reactions to spoken languages. The Journal of Abnormal and Social Psychology, 601, 44–51. Google Scholar logo with link to Google Scholar
Lappin-Fortin, K., & Rye, B. J. (2014). The use of pre-/posttest and self-assessment tools in a French pronunciation course. Foreign Language Annals, 471, 300–320. Google Scholar logo with link to Google Scholar
Lindemann, S. (2003). Koreans, Chinese or Indians? Attitudes and ideologies about non-native English speakers in the United States. Journal of Sociolinguistics, 71, 348–364. Google Scholar logo with link to Google Scholar
Major, R. C., Fitzmaurice, S. F., Bunta, F., & Balasubramanian, C. (2002). The effects of non-native accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 361, 173–190. Google Scholar logo with link to Google Scholar
Montgomery, C. (2007). Northern English dialects: A perceptual approach (Unpublished PhD dissertation). University of Sheffield, Sheffield, United Kingdom.Google Scholar logo with link to Google Scholar
Munro, M. J., Derwing, T. M., & Morton, S. L. (2006). The mutual intelligibility of L2 speech. Studies in Second Language Acquisition, 281, 111–131. Google Scholar logo with link to Google Scholar
Norton, B., & Toohey, K. (2011). Identity, language learning, and social change. Language Teaching, 441, 412–446. Google Scholar logo with link to Google Scholar
O’Brien, M. G. (2014). L2 learners’ assessments of accentedness, fluency, and comprehensibility of native and nonnative German speech. Language Learning, 641, 715–748. Google Scholar logo with link to Google Scholar
Pantos, A. J., & Perkins, A. W. (2013). Measuring implicit and explicit attitudes toward foreign accented speech. Journal of Language and Social Psychology, 321, 3–20. Google Scholar logo with link to Google Scholar
Preston, D. (1999). Handbook of perceptual dialectology 11. Philadelphia, PA: John Benjamins. Google Scholar logo with link to Google Scholar
Reid, K. T., Trofimovich, P., & O’Brien, M. G. (2019). Social attitudes and speech ratings: Effects of positive and negative bias on multiage listeners’ judgments of second language speech. Studies in Second Language Acquisition, 411, 419–442. Google Scholar logo with link to Google Scholar
Rose, R. L. (2017). Differences in second language speech fluency ratings: Native versus nonnative listeners. In L. Degand et al. (Eds.), Proceedings of the International Conference “Fluency & Disfluency Across Languages and Language Varieties” (pp. 101–103). Louvain-la-Neuve, Belgium: Université catholique de Louvain.Google Scholar logo with link to Google Scholar
Rossiter, M. J. (2009). Perceptions of L2 fluency by native and non-native speakers of English. Canadian Modern Language Review, 651, 395–412. Google Scholar logo with link to Google Scholar
Rubin, D. L. (1992). Nonlanguage factors affecting undergraduates’ judgments of nonnative English-speaking teaching assistants. Research in Higher Education, 331, 51–531. Google Scholar logo with link to Google Scholar
Ryan, E. B., Carranza, M. A., & Moffie, R. W. (1977). Reactions toward varying degrees of accentedness in the speech of Spanish-English bilinguals. Language and Speech, 201, 267–273. Google Scholar logo with link to Google Scholar
Ryan, E. B., Hewstone, M., & Giles, H. (1984). Language and intergroup attitudes. In J. R. Eiser (Ed.), Attitudinal judgment (pp. 135–158). New York: Springer. Google Scholar logo with link to Google Scholar
Sokolov, C. (2014). Self-evaluation of rater bias in written composition assessment. Linguistica, 541, 261–275. Google Scholar logo with link to Google Scholar
Spillane, J. P. (2006). Distributed leadership. San Francisco, CA: Jossey-Bass. Google Scholar logo with link to Google Scholar
Spillane, J. P., Hallett, T., & Diamond, J. B. (2003). Forms of capital and the construction of leadership: Instructional leadership in urban elementary schools. Sociology of Education, 761, 1–17. Google Scholar logo with link to Google Scholar
Steele, C. M., & Aronson, J. (1995). Stereotype threat and the intellectual test performance of African Americans. Journal of Personality and Social Psychology, 691, 797–811. Google Scholar logo with link to Google Scholar
Stewart, M. A., Ryan, E. G., & Giles, H. (1985). Accent and social class effects on status and solidarity evaluations. Personality and Social Psychology Bulletin, 111, 98–105. Google Scholar logo with link to Google Scholar
Tajfel, H. (1972). Experiments in a vacuum. In J. Israel & H. T. Triandis (Eds.), The context of social psychology: A critical assessment (pp. 69–119). London, UK: Academic Press.Google Scholar logo with link to Google Scholar
Isaacs, T., & Trofimovich, P. (2012). Deconstructing comprehensibility: Identifying the linguistic influences on listeners’ L2 comprehensibility ratings. Studies in Second Language Acquisition, 341, 475–505. Google Scholar logo with link to Google Scholar
van Bezooijen, R. (2002). Aesthetic evaluation of Dutch: Comparisons across dialects, accents and languages. In D. Long & D. Preston (Eds.), Handbook of perceptual dialectology (Vol. 21, pp. 13–30). Amsterdam, The Netherlands: John Benjamins. Google Scholar logo with link to Google Scholar
Visser, P. S., & Cooper, J. (2007). Attitude change. The Sage handbook of social psychology (pp. 197–218). Thousand Oaks, CA: Sage. Google Scholar logo with link to Google Scholar
Vissers, C. T. W. M., Virgillito, D., Fitzgerald, D. A., Speckens, A. E. M., Tendolkar, I., Van Oostrom, I., & Chwilla, D. J. (2010). The influence of mood on the processing of syntactic anomalies: Evidence from P600. Neuropsychologia, 481, 3521–3531. Google Scholar logo with link to Google Scholar
Watson, K., & Clark, L. (2015). Exploring listeners’ real-time reactions to regional accents. Language Awareness, 241, 38–59. Google Scholar logo with link to Google Scholar
Wheelan, S., & Kaeser, R. M. (1997). The influence of task type and designated leaders on developmental patterns in groups. Small Group Research, 281, 94–121. Google Scholar logo with link to Google Scholar
Winke, P. (2013). Rating oral language. In C. A. Chapelle (Ed.), The encyclopedia of applied linguistics (pp. 1–7). Hoboken, NJ: Blackwell. Google Scholar logo with link to Google Scholar
Winke, P., & Gass, S. (2013). The influence of second language experience and accent familiarity on oral proficiency rating: A qualitative investigation. TESOL Quarterly, 471, 762–789. Google Scholar logo with link to Google Scholar
Winke, P., Gass, S., & Myford, C. (2013). Raters’ L2 background as a potential source of bias in rating oral performance. Language Testing, 301, 231–252. Google Scholar logo with link to Google Scholar
Xu, W., Wang, Y., & Case, R. E. (2010). Chinese attitudes towards varieties of English: A pre-Olympic examination. Language Awareness, 191, 249–260. Google Scholar logo with link to Google Scholar
Yao, Z., Saito, K., Trofimovich, P., & Isaacs, T. (2013). Z-Lab [Computer software]. [URL]
Cited by (5)

Cited by five other publications

Levis, John M.
2024. Key issues in L2 pronunciation research. Journal of Second Language Pronunciation 10:3 ► pp. 293 ff. DOI logo
Teló, Cesar, Pavel Trofimovich, Mary Grantham O'Brien, Thao-Nguyen Nina Le & Anamaria Bodea
2024. Beyond the resume: HR students’ evaluations of interview performances by first and second language speakers. Equality, Diversity and Inclusion: An International Journal DOI logo
Brown, Bianca, Botagoz Tusmagambet, Valentino Rahming, Chun-Ying Tu, Michael B. DeSalvo & Seth Wiener
2023. Searching for the “native” speaker: A preregistered conceptual replication and extension of Reid, Trofimovich, and O’Brien (2019). Applied Psycholinguistics 44:4 ► pp. 475 ff. DOI logo
Taylor Reid, Kym, Pavel Trofimovich & Mary Grantham O’Brien
2022. The Malleability of Listener Judgments of Second Language Speech. In Theoretical and Practical Developments in English Speech Assessment, Research, and Training [Second Language Learning and Teaching, ], ► pp. 11 ff. DOI logo
Trofimovich, Pavel, Oguzhan Tekin & Kim McDonough
2021. Task engagement and comprehensibility in interaction. Journal of Second Language Pronunciation 7:3 ► pp. 435 ff. DOI logo

This list is based on CrossRef data as of 13 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue