Cover not available

Article published In: Pragmatics
Vol. 21:3 (2011) ► pp.453–471

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF

Rater variation in the assessment of speech acts

Open Access logo
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.
Published online: 1 September 2011
DOI logo with link
https://doi.org/10.1075/prag.21.3.08tag
This study addresses variability among native speaker raters who evaluated pragmatic performance of learners of English as a foreign language. Using a five-point rating scale, four native English speakers of mixed cultural background (one African American, one Asian American, and two Australians) assessed the appropriateness of two types of speech acts (requests and opinions) produced by 48 Japanese EFL students. To explore norms and the reasoning behind the raters’ assessment practice, individual introspective verbal interviews were conducted. Eight students’ speech act productions (64 speech acts in total) were selected randomly, and the raters were asked to rate each speech act and then explain their rating decision. Interview data revealed similarities and differences in their use of pragmatic norms and social rules in evaluating appropriateness.
References (31)
Austin, J.L. (1962) How to do things with words. Cambridge: Harvard University Press.Google Scholar logo with link to Google Scholar
Bachman, Lyle, and Adrian Palmer (1996) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Brown, Annie (2000) An investigation of rater's orientation in awarding scores in the IELTS interview. In R. Tulloch (ed.), IELTS Research Report 31: 49-84.Google Scholar logo with link to Google Scholar
(2003) Interviewer variation and the co-construction of speaking proficiency. Language Testing 201: 1-25. Google Scholar logo with link to Google Scholar
(2005) Interviewer Variability in Oral Proficiency Interviews. Frankfurt am Main: Peter Lang.Google Scholar logo with link to Google Scholar
Brown, James D. (2001) Six types of pragmatics tests in two different contexts. In K. Rose & G. Kasper (eds.), Pragmatics in Language Teaching. Cambridge: Cambridge University Press, pp. 301-325. Google Scholar logo with link to Google Scholar
Clark, H.H. (1979) Responding to indirect speech acts. Cognitive Psychology 111: 430-477. Google Scholar logo with link to Google Scholar
Cohen, Andrew (1994) Assessing Language Ability in the Classroom. Rowley, MS: Newbury House.Google Scholar logo with link to Google Scholar
Cohen, Andrew, and Elite Olshtain (1981) Developing a measure of socio-cultural competence: The case of apology. Language Learning 311: 113-134. Google Scholar logo with link to Google Scholar
Davies, Alan (2003) The native speaker in applied linguistics. In A. Davies & K. Elder (eds.), Handbook of Applied Linguistics. New York: Blackwell, pp. 431-450.Google Scholar logo with link to Google Scholar
Ducassee, Ana M. (2009) Assessing paired orals: Raters' orientation to interaction. Language Testing 261: 423-443. Google Scholar logo with link to Google Scholar
Green, A. (1998) Verbal Protocol Analysis in Language Testing Research: A Handbook (vol.51). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Hoffman-Hicks, Sheila (1992) Linguistic and pragmatic competence: Their relationship in the overall competence of the language learner. In L.F. Bouton & Y. Kachru (eds.), Pragmatics and Language Learning Monograph Series Vol.31.Urbana-Champaign, IL: University of Illinois, pp. 66-80.Google Scholar logo with link to Google Scholar
Hudson, Thom, Emily Detmer, and James D. Brown (1995) Developing Prototypic Measures of Cross- Cultural Pragmatics (Technical Report No.7). Honolulu, HI: University of Hawai'I at Manoa, Second Language Teaching & Curriculum Center.Google Scholar logo with link to Google Scholar
Johnson, Jeff, and Gad Lim (2009) The influence of rater language background on writing performance assessment. Language Testing 261: 485-505. Google Scholar logo with link to Google Scholar
Levinson, Stephen (1983) Pragmatics. Cambridge: Cambridge University Press.  BoP. Google Scholar logo with link to Google Scholar
Liu, Jianda (2006) Measuring Interlanguage Pragmatic Knowledge of EFL Learners. Frankfurt: Peter Lang.Google Scholar logo with link to Google Scholar
May, Lyn (2007) Interaction in a paired speaking test: The rater's perspective. Ph.D. dissertation: The University of Melbourne.
(2009) Co-constructed interaction in a paired speaking test: The rater's perspective. Language Testing 261: 397-421. Google Scholar logo with link to Google Scholar
McNamara, Tim, and Tom Lumley (1997) The effect of interlocutor and assessment mode variables in overseas assessments of speaking skills in occupational settings. Language Testing 141: 140-156. Google Scholar logo with link to Google Scholar
Mey, Jacob (1993) Pragmatics: An Introduction. Oxford: Blackwell.  BoPGoogle Scholar logo with link to Google Scholar
Roever, Carston (2005) Testing EFL Pragmatics. Frankfurt: Peter Lang. Google Scholar logo with link to Google Scholar
O'Loughlin, Kieran (2002) The impact of gender in oral proficiency testing. Language Testing 191: 169-192. Google Scholar logo with link to Google Scholar
Poliitt, Alastair, and Neil Murray (1996) What do raters really pay attention to? In M. Milanovic & N. Saville (eds.), Performance Testing Cognition and Assessment: Selected papers from the 15th Language Testing Research Colloquium. Cambridge: Cambridge University Press, pp. 74-91.Google Scholar logo with link to Google Scholar
Thomas, Jenny (1995) Meaning in Interaction: An Introduction to Pragmatics. London: Longman.  BoPGoogle Scholar logo with link to Google Scholar
Sasaki, Miyuki (1998) Investigating EFL students' production of speech acts: A comparison of production questionnaires and role-plays. Journal of Pragmatics 301: 457-484.  BoPGoogle Scholar logo with link to Google Scholar
Walter, Scott (2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing 241: 155-183. Google Scholar logo with link to Google Scholar
Yamashita, Sayoko (1996) Six Measures of JSL Pragmatics (Technical report #14). Honolulu, HI: University of Hawai'i at Manoa, Second Language Teaching & Curriculum Center.Google Scholar logo with link to Google Scholar
(2001) Using pictures for research in pragmatics-eliciting pragmatic strategies by picture response tests. In T. Hudson & J.D. Brown (eds.), A Focus on Language Test Development: Expanding the Language Proficiency Construct Across a Variety of Tests. (Technical Report #21). University of Hawaii, Second language Teaching and Curriculum Center, pp. 35-56.Google Scholar logo with link to Google Scholar
Yoshitake, S. (1997) Measuring interlanguage pragmatic competence of Japanese students of English as a foreign language: A multi-test framework evaluation. Unpublished Ph.D. thesis, Columbia Pacific University, Novata, CA.
Cited by (22)

Cited by 22 other publications

Alemi, Minoo & Neda Khanlarzadeh
2024. Input-Providing and Output-Pushing E-Feedback in L2 Pragmatics. In The Handbook of Cultural Linguistics [Springer Handbooks in Languages and Linguistics, ], ► pp. 849 ff. DOI logo
Dixon, Tülay & Daniel H. Dixon
2024. A simple methodology for identifying speech act tendencies. Applied Pragmatics 6:1 ► pp. 60 ff. DOI logo
Kostromitina, Maria & Yongzhi Miao
2024. Listener perception of appropriateness of L1 and L2 refusals in English. Studies in Second Language Learning and Teaching 14:2 ► pp. 291 ff. DOI logo
Li, Shuai, Ting Wen, Xian Li, Yali Feng & Chuan Lin
2023. Comparing holistic and analytic marking methods in assessing speech act production in L2 Chinese. Language Testing 40:2 ► pp. 249 ff. DOI logo
Beaulieu, Suzie, Fanny Forsberg Lundell & Javier Bejarano
2022. Interlocutors’ judgment of Lx conventional expressions: An exploratory study. Intercultural Pragmatics 19:5 ► pp. 597 ff. DOI logo
Cheng, Dongmei
2022. “Communication is a two-way street”. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) ► pp. 1 ff. DOI logo
Gesuato, Sara & Victoriya Trubnikova
2022. Assessing Pragmatic Skills in Speech Act Production and Perception. In Handbook of Research on Policies and Practices for Assessing Inclusive Teaching and Learning [Advances in Educational Marketing, Administration, and Leadership, ], ► pp. 211 ff. DOI logo
Omar, Fouad Rashid & Özge Razı
2022. Impact of instruction based on movie and TV series clips on EFL learners’ pragmatic competence: Speech acts in focus. Frontiers in Psychology 13 DOI logo
SHLEYKİNA, Galina
2021. Pragmatic failure revisited: Jaworski’s (1994) study in a new light. Eurasian Journal of Applied Linguistics ► pp. 303 ff. DOI logo
Sickinger, Pawel & Katrin Renkwitz
2021. How Good Are Native Speakers at Evaluating Pragmatic Competence? Native Speaker Assessments of Learner Status and Appropriate Pragmatic Behaviour in English. Langages N° 222:2 ► pp. 113 ff. DOI logo
Arnold, Nike
2020. Community, Group Dynamics, Identity, and Anxiety in Learning an Additional Language. In The Cambridge Introduction to Applied Linguistics, ► pp. 71 ff. DOI logo
Susan Conrad, Alissa Hartig & Lynn Santelmann
2020. The Cambridge Introduction to Applied Linguistics, DOI logo
Lu, Xiaofei & Brody Bluemel
2020. Automated Assessment of Language. In The Cambridge Introduction to Applied Linguistics, ► pp. 86 ff. DOI logo
Sonnenburg-Winkler, Sunni L., Zohreh R. Eslami & Ali Derakhshan
2020. Rater variation in pragmatic assessment: The impact of the linguistic background on peer-assessment and self-assessment. Lodz Papers in Pragmatics 16:1 ► pp. 67 ff. DOI logo
Li, Shuai, Naoko Taguchi & Feng Xiao
2019. Variations in Rating Scale Functioning in Assessing Speech Act Production in L2 Chinese. Language Assessment Quarterly 16:3 ► pp. 271 ff. DOI logo
Pinto, Derrin
2019. Shifting responsibilities: Student e-mail excuses and how faculty perceive them. Lingua 222 ► pp. 53 ff. DOI logo
Hernandez, Todd A.
2016. Acquisition of L2 Spanish requests in short-term study abroad. Study Abroad Research in Second Language Acquisition and International Education 1:2 ► pp. 186 ff. DOI logo
Kusevska, Marija, Biljana Ivanovska, Nina Daskalovska & Tatjana Ulanska
2016. Profiling Pragmatic Competence of Foreign Language Learners. Romanian Journal of English Studies 13:1 ► pp. 79 ff. DOI logo
Alemi, Minoo, Zohreh R. Eslami & Atefeh Rezanejad
2014. Rating EFL Learners’ Interlanguage Pragmatic Competence by Non-Native English Speaking Teachers. Procedia - Social and Behavioral Sciences 98 ► pp. 171 ff. DOI logo
Tajeddin, Zia & Minoo Alemi
2014. Criteria and Bias in Native English Teachers’ Assessment of L2 Pragmatic Appropriacy: Content and FACETS Analyses. The Asia-Pacific Education Researcher 23:3 ► pp. 425 ff. DOI logo
Tajeddin, Zia & Minoo Alemi
2020. Pragmatics and Good Language Teachers. In Lessons from Good Language Teachers, ► pp. 189 ff. DOI logo
Ishihara, Noriko
2013. Teacher-based assessment of L2 Japanese pragmatics: Classroom applications. In Assessing Second Language Pragmatics, ► pp. 124 ff. DOI logo

This list is based on CrossRef data as of 30 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue