Cover not available

In:Instructed Second Language Acquisition Research Methods
Edited by Laura Gurzynski-Weiss and YouJin Kim
[Research Methods in Applied Linguistics 3] 2022
► pp. 149–180

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 7
Pragmatics

Assessing learning outcomes in instructional studies

 | Northern Arizona University
 | Daegu National University of Education, Korea
Published online: 8 December 2022
DOI logo with link
https://doi.org/10.1075/rmal.3.07tag
Abstract
The field of instructed SLA (ISLA) has grown rapidly in recent years to examine how systematic manipulations of instructional conditions can lead to the development of second language (L2) knowledge and use (Loewen & Sato, 2017). Following this trend, L2 pragmatics researchers have implemented various instructional methods and examined their effectiveness using experimental designs (Taguchi, 2015; Taguchi & Roever, 2017). The critical issue in this practice is how to assess learning outcomes in order to make a claim that certain instructional methods can produce robust pragmatic knowledge. To address this question, this chapter presents an overview of assessment methods used in analyzing pragmatics learning outcomes. The chapter surveys instructed L2 pragmatics studies published in the last four decades to identify common assessment methods (i.e., discourse completion tasks and role-play tasks). We provide step-by-step illustrations of expert studies in order to demonstrate how researchers design an assessment task, evaluate learning, and interpret results. The chapter concludes with a discussion of the challenges and limitations of current methods and provides direction for addressing such challenges in pragmatics research.
Article outline
  • 1.Introduction: What is pragmatics and why is it important?
  • 2.What we know and what we need to know about pragmatics in ISLA
    • 2.1Current trends in instructed pragmatics
  • 3.Data elicitation and interpretation options for pragmatics
    • 3.1Discourse completion tasks (DCTs)
      • 3.1.1Characteristics of DCTs
      • 3.1.2Pros and cons of DCTs
      • 3.1.3Step-by-step DCT illustration
    • 3.2Role-play tasks
      • 3.2.1Characteristics of role-play tasks
      • 3.2.2Pros and cons of role-play tasks
      • 3.2.3Step-by-step role play illustration
  • 4.Advice to future pragmatics researchers in ISLA: How to improve assessment methods of learning outcomes
    • 4.1Incorporating prosody as instructional targets and learning outcomes
    • 4.2Improving authenticity of DCTs and role-plays via technology applications
    • 4.3Enhancing consequentiality in role-play interaction
  • 5.Troubleshooting ISLA pragmatics research
    • Common question 1: How can we choose the most appropriate method to assess learning outcomes? Should we choose DCTs or role-plays?
    • Common question 2: How can we determine assessment criteria and score bands when evaluating learning outcomes?
  • 6.Conclusions
  • 7.Further reading and additional resources
    • 7.1Suggested books
    • 7.2Suggested journals, professional organizations, and websites
  • Note
  • References
  • Appendix
References (47)
References
Alemi, M., & Haeri, N. S. (2020). Robot-assisted instruction of L2 pragmatics: Effects on young EFL learners’ speech act performance. Language Learning & Technology, 24(2), 86–103.Google Scholar logo with link to Google Scholar
Al-Gahtani, S., & Roever, C. (2012). Proficiency and sequential organization of L2 requests. Applied Linguistics, 33(1), 42–65. Google Scholar logo with link to Google Scholar
(2018). Proficiency and preference organization in second language refusals. Journal of Pragmatics, 129, 140–153. Google Scholar logo with link to Google Scholar
Barón, J., Celaya, M. L., & Levkina, M. (2020). Learning pragmatics through tasks: When interaction plays a role. Applied Pragmatics, 2(1), 1–25. Google Scholar logo with link to Google Scholar
Blum-Kulka, S., House, J., & Kasper, G. (1989). Cross-cultural pragmatics: Requests and apologies. Ablex.Google Scholar logo with link to Google Scholar
Boersma, P., & Weenink, D. (2016). Praat: Doing phonetics by computer (Version 6.0.11) [Software]. Available from [URL]
Chen, S. (2019, October 4–5). Developing an L2 pragmatic speaking test using conversation analysis findings [Paper presentation]. Annual Midwest Association of Language Testers (MwALT) conference, Indiana University Bloomington.
Cohen, A. D., & Shively, R. L. (2007). Acquisition of requests and apologies in Spanish and French: Impact of study abroad and strategy-building intervention. Modern Language Journal, 91(2), 189–212. Google Scholar logo with link to Google Scholar
Eslami, Z. R., Mirzaei, A., & Dini, S. (2015). The role of asynchronous computer mediated communication in the instruction and development of EFL learners’ pragmatic competence. System, 48, 99–111. Google Scholar logo with link to Google Scholar
Félix-Brasdefer, J. C. (2007). Pragmatic development in the Spanish as a FL classroom: A cross-sectional study of learner requests. Intercultural Pragmatics, 4(2), 253–286. Google Scholar logo with link to Google Scholar
Golato, A. (2003). Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24(1), 90–121. Google Scholar logo with link to Google Scholar
Grabowski, K. C. (2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking [Unpublished doctoral dissertation]. Columbia University.
Gumperz, J. J. (1982). Discourse strategies. Cambridge University Press. Google Scholar logo with link to Google Scholar
Gumperz, J. J., & Cook-Gumperz, J. (1982). Introduction: Language and the communication of social identity. In J. J. Gumperz (Ed.), Language and social identity (pp. 1–21). Cambridge University Press.Google Scholar logo with link to Google Scholar
Halenko, N. (2013). Using computer animation to assess and improve spoken language skills. In Conference proceedings: ICT for language learning (p. 286). Libreriauniversitaria.Google Scholar logo with link to Google Scholar
Halenko, N., & Jones, C. (2017). Explicit instruction of spoken requests: An examination of pre-departure instruction and the study abroad environment. System, 68, 26–37. Google Scholar logo with link to Google Scholar
Hashimoto, B. J., & Nelson, K. (2020). Using a corpus in creating and evaluating a DCT. Applied Pragmatics, 2(1), 80–120. Google Scholar logo with link to Google Scholar
Hassall, T. (2020). Preference structure in request sequences: What about role-play? Journal of Pragmatics, 155, 321–333. Google Scholar logo with link to Google Scholar
Huth, T. (2010). Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal, 94(4), 537–553. Google Scholar logo with link to Google Scholar
Kang, O., Kermad, A., & Taguchi, N. (2021). The interplay of proficiency and study abroad experience on prosody of L2 speech acts. Journal of Second Language Pronunciation, 7(3), 343–369. Google Scholar logo with link to Google Scholar
(2008). Data collection in pragmatics research. In H. Spencer-Oatey (Ed.), Culturally speaking: Culture, communication and politeness theory (pp. 279–302). Continuum.Google Scholar logo with link to Google Scholar
Kasper, G., & Dahl, M. (1991). Research methods in interlanguage pragmatics. Studies in Second Language Acquisition, 13(2), 215–247. Google Scholar logo with link to Google Scholar
Kasper, G., & Youn, S. J. (2018). Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review, 9(4), 589–616. Google Scholar logo with link to Google Scholar
Kim, M., Lee, H., & Kim, Y. (2018). Learning of Korean honorifics through collaborative tasks: Comparing heritage and foreign language learners. In N. Taguchi & Y. Kim (Eds.), Task-based approaches to teaching and assessing pragmatics (pp. 27–54). John Benjamins. Google Scholar logo with link to Google Scholar
Li, S. (2012). The effects of input-based practice on pragmatic development of requests in L2 Chinese. Language Learning, 62(2), 403–438. Google Scholar logo with link to Google Scholar
Loewen, S., & Sato, M. (Eds.). (2017). The Routledge handbook of instructed second language acquisition. Routledge. Google Scholar logo with link to Google Scholar
Nguyen, H. T. (2018). Interactional practices across settings: From classroom role-plays to workplace patient consultations. Applied Linguistics, 39(2), 213–235.Google Scholar logo with link to Google Scholar
Okada, Y. (2010). Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42(6), 1647–1668. Google Scholar logo with link to Google Scholar
Plonsky, L., & Zhuang, J. (2019). A meta-analysis of second language pragmatics instruction. In N. Taguchi (Ed.), Routledge handbook of SLA and pragmatics (pp. 287–307). Routledge.Google Scholar logo with link to Google Scholar
Pomerantz, A., & Heritage, J. (2013). Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Wiley-Blackwell.Google Scholar logo with link to Google Scholar
Rockey, C., Tiegs, J., & Fernández, F. (2020). Mobile application use in technology-enhanced DCTs. CALICO, 37(1), 85–108. Google Scholar logo with link to Google Scholar
Schmidt, R. (1993). Consciousness, learning and interlanguage pragmatics. In G. Kasper & S. Blum-Kulka (Eds.), Interlanguage pragmatics (pp. 21–42). Oxford University Press.Google Scholar logo with link to Google Scholar
Stokoe, E. (2013). The (in)authenticity of simulated talk: Comparing role-played and actual interaction and the implications for communication training. Research on Language and Social Interaction, 46(2), 165–185. Google Scholar logo with link to Google Scholar
Swain, M., & Lapkin, S. (1998). Interaction and second language learning: Two adolescent French immersion students working together. Modern Language Journal, 82(1), 320–337. Google Scholar logo with link to Google Scholar
Sydorenko, T., Jones, Z. W., Daurio, P., & Thorne, S. L. (2020). Beyond the curriculum: Extended discourse practice through self-access pragmatics simulations. Language Learning & Technology, 24(2), 48–69.Google Scholar logo with link to Google Scholar
Sykes, J. M., & González-Lloret, M. (2020). Exploring the interface of interlanguage (L2) pragmatics and digital spaces. CALICO Journal, 37(1), i–xv. Google Scholar logo with link to Google Scholar
Taguchi, N. (2015). Instructed pragmatics at a glance: Where instructional studies were, are, and should be going. Language Teaching, 48(1), 1–50. Google Scholar logo with link to Google Scholar
(2021). Application of immersive virtual reality (VR) to pragmatics data collection methods: Insights from interviews. CALICO, 38(2), 181–201. Google Scholar logo with link to Google Scholar
Taguchi, N., Hirschi, K., & Kang, O. (2021). Longitudinal L2 development in the prosodic marking of pragmatic meaning: Prosodic changes in L2 speech acts and individual factors. Studies in Second Language Acquisition, 1–16. Advance online publication. Google Scholar logo with link to Google Scholar
Taguchi, N., & Kim, Y. (2016). Collaborative dialogue in learning pragmatics: Pragmatic-related episodes as an opportunity for learning request-making. Applied Linguistics, 37(3), 416–437. Google Scholar logo with link to Google Scholar
Taguchi, N., Li, Q., & Tang, X. (2017). Learning Chinese formulaic expressions in a scenario-based interactive environment. Foreign Language Annals, 50(4), 641–660. Google Scholar logo with link to Google Scholar
Taguchi, N., & Roever, C. (2017). Second language pragmatics. Oxford University Press.Google Scholar logo with link to Google Scholar
Takimoto, M. (2020). Investigating the effects of cognitive linguistic approach in developing EFL learners’ pragmatic proficiency. System, 89. Google Scholar logo with link to Google Scholar
Verschueren, J. (1999). Understanding pragmatics. Oxford University Press.Google Scholar logo with link to Google Scholar
Youn, S. J. (2020). Pragmatic variables in role-play design for the context validity of assessing interactional competence. Papers in Language Testing and Assessment, 9(1), 95–127.Google Scholar logo with link to Google Scholar
Cited by (2)

Cited by two other publications

Taguchi, Naoko
2024. Technology-enhanced language learning and pragmatics: Insights from digital game-based pragmatics instruction. Language Teaching 57:1 ► pp. 57 ff. DOI logo
Kim, YouJin, Sanghee Kang, Meredith D’Arienzo & Naoko Taguchi
2023. Comparing traditional and task-based approaches to teaching pragmatics: Task design processes and learning outcomes. Language Teaching Research DOI logo

This list is based on CrossRef data as of 30 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue