Cover not available

In:Task-Based Approaches to Teaching and Assessing Pragmatics
Edited by Naoko Taguchi and YouJin Kim
[Task-Based Language Teaching 10] 2018
► pp. 217–246

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 9
Task design and validity evidence for assessment of L2 pragmatics in interaction

 | Northern Arizona University
Published online: 15 August 2018
DOI logo with link
https://doi.org/10.1075/tblt.10.09you
Abstract
This study examines how clear communicative goals and authentic interaction are ensured in task-based pragmatic assessment practices, particularly in designing role-play assessment tasks and developing task-appropriate rating criteria. Employing a concurrent mixed methods design, conversation analysis (CA) and multi-faceted Rasch measurement were used to investigate whether task-independent interactional features are elicited from role-play interactions and how interaction-specific rating categories function quantitatively. The CA findings indicated that various interactional organizations emerged from role-play task interactions as examinees oriented to the different degree of imposition and contextual variables embedded in the role-plays. Further, the interactional features included in the rating criteria created a substantial amount of variance in distinguishing varying degrees of pragmatic performance. The findings are discussed in terms of designing valid pragmatic assessment tasks and what features need to be included in developing rating criteria to ensure the validity evidence of task-based pragmatic assessment.
Article outline
  • Introduction
  • Background
    • Assessing pragmatic competence in interaction and validity challenges
    • Role-plays as assessment tasks and eliciting validity evidence
    • The present study
  • Methods
    • Participants
      • Examinees
      • Interlocutors
      • Raters
    • Test instruments
      • Role-plays
      • Rating criteria
    • Procedures
      • Test administration
      • Rater training
      • Rating
    • Data analysis
      • Conversation analysis
      • Multi-faceted Rasch measurement
  • Results
    • Conversation Analysis (CA) findings
      • Recommendation letter request role-play
      • Additional advising time request role-play
      • Refusing a professor’s request role-play
      • Deciding a meeting time
      • Deciding a meeting mode
    • Multi-faceted Rasch Measurement analysis
  • Discussion
  • Conclusion
  • References
  • Appendix
References (46)
References
Al-Gahtani, S., & Roever, C. (2012). Proficiency and sequential organization of L2 requests. Applied Linguistics, 33, 42–65. Google Scholar logo with link to Google Scholar
(2014). Insert and post-expansion in L2 Arabic requests. System, 42, 189–206. Google Scholar logo with link to Google Scholar
Atkinson, J., & Heritage, J. (Eds.) (1984). Structures of social action. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., & Hartford, B. S. (Eds.) (2005). Interlanguage pragmatics: Exploring institutional talk. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model (2nd ed.). New York, NY: Routledge.Google Scholar logo with link to Google Scholar
Celce-Murcia, M. (2007). Rethinking the role of communicative competence in language teaching. In E. Alcón Soler & M. P. Safont Jordà (Eds.), Intercultural language use and language learning (pp. 41–57). Dordrecht: Springer. Google Scholar logo with link to Google Scholar
Celce-Murcia, M., Dörnyei, Z., & Thurrell, S. (1995). Communicative competence: A pedagogically motivated model with content specifications. Issues in Applied Linguistics, 6, 5–35.Google Scholar logo with link to Google Scholar
Chapelle, C. A. (2012). Validity argument for language assessment: The framework is simple… Language Testing, 29, 19–27. Google Scholar logo with link to Google Scholar
Chapelle, C., Enright, M., & Jamieson, J. (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. New York, NY: Routledge.Google Scholar logo with link to Google Scholar
Eckes, T. (2011). Introduction to many-Facet Rasch measurement. Frankfurt: Peter Lang.Google Scholar logo with link to Google Scholar
Grabowski, K. (2013). Investigating the construct validity of a role-play test designed to measure grammatical and pragmatic knowledge at multiple proficiency levels. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 149–171). Houndmills: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Heritage, J. (1984). A change-of-state token and aspects of its sequential placement. In M. J. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 299–345). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Hudson, T. D., Detmer, E., & Brown, J. D. (1992). A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.Google Scholar logo with link to Google Scholar
(1995). Developing prototype measures of cross-cultural pragmatics (Technical Report #7). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.Google Scholar logo with link to Google Scholar
Hughes, A. (1989). Testing for language teachers. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Huth, T. (2010). Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal, 94, 537–553. Google Scholar logo with link to Google Scholar
Kane, M. (1992). An argument-based approach to validity. Psychological Bulletin, 112, 527–535. Google Scholar logo with link to Google Scholar
(2006). Validation. In R. L. Brennan (Ed.), Educational measurement (pp. 17–64). Westport, CT: American Council on Education.Google Scholar logo with link to Google Scholar
Kasper, G. (2006). Speech acts in interaction: Towards discursive pragmatics. In K. Bardovi-Harlig, C. Félix-Brasdefer & A. S. Omar (Eds.), Pragmatics and language learning (pp. 281–314). Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawai‘i.Google Scholar logo with link to Google Scholar
Kasper, G., & Rose, K. R. (2002). Pragmatic development in a second language. Malden, MA: Blackwell Publishing.Google Scholar logo with link to Google Scholar
Kasper, G. (2009). L2 pragmatic development. In W. C. Ritchie & T. K. Bhatia (Eds.), New handbook of second language acquisition (pp. 259–295). Leeds: Emerald.Google Scholar logo with link to Google Scholar
Kasper, G., & Ross, S. (2013). Assessing second language pragmatics: An overview and introductions. In S. J. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 1–40). Houndmills: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Kasper, G., & Youn, S. J. (2017). Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review. Advance online publication. Google Scholar logo with link to Google Scholar
Linacre, J. M. (2014). Facets Rasch measurement computer program, version 3.71.4. Chicago, IL: Winsteps.com.Google Scholar logo with link to Google Scholar
Liu, J. (2007). Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391–415. Google Scholar logo with link to Google Scholar
Lunz, M. E., Wright, B. D., & Linacre, J. M. (1990). Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331–345. Google Scholar logo with link to Google Scholar
McNamara, T. (1996). Measuring second language performance. New York, NY: Addison Wesley Longman.Google Scholar logo with link to Google Scholar
McNamara, T., & Knoch, U. (2012). The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29, 555–576. Google Scholar logo with link to Google Scholar
Norris, J. M. (2009). Task-based teaching and testing. In M. H. Long & C. J. Doughty (Eds.), Handbook of language teaching (pp. 578–594). Oxford: Blackwell. Google Scholar logo with link to Google Scholar
(2016). Current uses for task-based language assessment. Annual Review of Applied Linguistics, 36, 230–244.Google Scholar logo with link to Google Scholar
Okada, Y. (2010). Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42, 1647–1668. Google Scholar logo with link to Google Scholar
Okada, Y., & Greer, T. (2013). Pursuing a relevant response in oral proficiency interview role plays. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 288–310). Houndmills: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Pomerantz, A. (1984). Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 57–101). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Pomerantz, A., & Heritage, J. (2013). Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Chichester: Wiley-Blackwell.Google Scholar logo with link to Google Scholar
Purpura, J. E., Brown, J. D., & Schoonen, R. (2015). Improving the validity of quantitative measures in applied linguistics research. Language Learning, 65, 37–75. Google Scholar logo with link to Google Scholar
Roever, C. (2006). Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229–256. Google Scholar logo with link to Google Scholar
(2011). Testing of second language pragmatics: Past and future. Language Testing, 28, 463–481. Google Scholar logo with link to Google Scholar
Ross, S. J., & Kasper, G. (Eds.) (2013). Assessing second language pragmatics. Houndmills: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Sacks, H. (1992). Lectures on conversation (2 Vols.). Oxford: Blackwell.Google Scholar logo with link to Google Scholar
Schegloff, E. A. (2007). Sequence organization in interaction: A primer in conversation analysis. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Schumacker, R. E. (1999). Many-facet Rasch analysis with crossed, nested, and mixed designs. Journal of Outcome Measurement, 3, 323–338.Google Scholar logo with link to Google Scholar
Taguchi, N., & Roever, C. (2017). Second language pragmatics. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Teddlie, C., & Tashakkori, A. (2006). A general typology of research designs featuring mixed methods. Research in the Schools, 13(1), 12–28.Google Scholar logo with link to Google Scholar
Walters, F. S. (2007). A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing, 24, 155–183. Google Scholar logo with link to Google Scholar
Youn, S. J. (2010). Task-based needs analysis of L2 pragmatics in an EAP context: Implications for teaching. Unpublished manuscript, University of Hawai‘i.Google Scholar logo with link to Google Scholar
(2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32, 199–225. Google Scholar logo with link to Google Scholar
Cited by (5)

Cited by five other publications

Asseldonk, Anne G. H. van, Christine C. F. Troussart Van Bulck, Lies Strobbe & Goedele Vandommele
2025. The development and reception of a digital task-based exam for young learners of Dutch as a Foreign Language. TASK. Journal on Task-Based Language Teaching and Learning 5:1 ► pp. 25 ff. DOI logo
Róg, Tomasz
2025. Empirical Investigations into the Use of Tasks in L2 Pragmatics Instruction. In Task-based Language Teaching and Beyond: L2 Pragmatics Instruction for Young Learners [Second Language Learning and Teaching, ], ► pp. 143 ff. DOI logo
Róg, Tomasz
2025. Pragmatics in Second Language Acquisition: Overview of Selected Issues. In Task-based Language Teaching and Beyond: L2 Pragmatics Instruction for Young Learners [Second Language Learning and Teaching, ], ► pp. 1 ff. DOI logo
Su, Yunwen & Sun-Young Shin
2024. Comparing two formats of data-driven rating scales for classroom assessment of pragmatic performance with roleplays. Language Testing 41:2 ► pp. 357 ff. DOI logo
González-Lloret, Marta
2022. The present and future of functional adequacy. TASK. Journal on Task-Based Language Teaching and Learning 2:1 ► pp. 146 ff. DOI logo

This list is based on CrossRef data as of 5 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue