Cover not available

Article published In:
[Applied Pragmatics 5:1] 2023
► pp. 41–63

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Validating a test of L2 routine formulae to detect pragmatics learning in stay abroad

ORCiD logo with link | The University of Melbourne
 | Sophia University
ORCiD logo with link | Waseda University
 | Kansai University
 | Keio University
Published online: 4 March 2022
DOI logo with link
https://doi.org/10.1075/ap.20015.roe
Abstract
There has long been interest in the effect of stay abroad on pragmatic development in second language (L2) pragmatics research. However, few testing instruments exist to assess pragmatics learning in stay abroad and provide information about learners’ development to help institutions evaluate the success of stay abroad programs. In this study, we describe the adaptation and validation of Roever, C. (2005). Testing ESL pragmatics. Peter Lang. Google Scholar logo with link to Google Scholar test of second language routine formulae to be used large-scale with Japanese stay-abroad learners. We follow Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.) (pp. 17–64). American Council on Education/Praeger Publishers.Google Scholar logo with link to Google Scholar argument-based approach to validation, evaluating the first four inferences: Domain description, Evaluation, Generalization, and Explanation. We also investigate the test’s practicality within the institutional setting of intended use. We find evidence supporting the use of the test for providing information on pragmatics learning during stay abroad. The test can differentiate between low- and high-ability learners, shows adequate reliability, and reflects the roles of proficiency and exposure in pragmatics learning in stay abroad contexts. The test is also practical and requires few resources. Further steps, such as the investigation of the Extrapolation and Decision inference, are discussed.
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Pragmatic development in study abroad: The case of routine formulae
    • 2.2Past tests of routine formula recognition and the argument-based approach
  • 3.This study
  • 4.Method
    • 4.1Participants
    • 4.2Instruments
    • 4.3Procedures
    • 4.4Data analysis
  • 5.Results
    • 5.1RQ1 Practicality: What is the cost of creating, administering, and scoring the test?
    • 5.2RQ2 Domain description: Do items sample the domain of routine formula knowledge sufficiently based on expert judgment?
    • 5.3RQ3 evaluation: Can item scoring be argued to follow clear procedures, is computation of total scores defensible, and does item difficulty spread from easy to difficult?
    • 5.4RQ4 generalization: Is the sample of items representative of the universe of generalization based on acceptable Cronbach’s alpha reliability?
    • 5.5Explanation
      • 5.5.1RQ5a Criterion: Does the test correlate moderately to highly with a general proficiency measure like TOEFL?
      • 5.5.2RQ5b group differences 1: Does recognition of routine formulae increase with length of stay abroad?
      • 5.5.3RQ5c explanation: Group differences 2: Do test-takers with stay abroad outperform a proficiency-matched group without stay abroad?
  • 6.Discussion
  • 7.Conclusion
  • Acknowledgements
  • References
References (54)
References
Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge University Press. Google Scholar logo with link to Google Scholar
Bachman, L. F., & Palmer, A. S. (2010). Language assessment in practice: Developing language tests and justifying their use in the real world. Oxford University Press.Google Scholar logo with link to Google Scholar
Baker-Smemoe, W., Dewey, D. P., Bown, J., & Martinsen, R. A. (2014). Variables affecting L2 gains during study abroad. Foreign Language Annals, 47(3), 464–486. Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K. (2009). Conventional expressions as a pragmalinguistic resource: Recognition and production of conventional expressions in L2 pragmatics. Language Learning, 59(4), 755–795. Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., & Bastos, M. T. (2011). Proficiency, length of stay, and intensity of interaction and the acquisition of conventional expressions in L2 pragmatics. Intercultural Pragmatics, 81, 347–384. Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? Pragmatic vs. grammatical awareness in instructed L2 learning. TESOL Quarterly, 321, 233–259. Google Scholar logo with link to Google Scholar
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E. (2015). The effect of instruction on pragmatic routines in academic discussion. Language Teaching Research, 19(3), 324–350. Google Scholar logo with link to Google Scholar
Barron, A. (2019). Pragmatic development and stay abroad. Journal of Pragmatics, 1461, 43–53. Google Scholar logo with link to Google Scholar
Brecht, R., Davidson, D., & Ginsberg, R. (1995). Predictors of foreign language gain during study abroad. In B. F. Freed (Ed.), Second language acquisition in a study abroad context (pp. 37–66). John Benjamins. Google Scholar logo with link to Google Scholar
Chapelle, C. A. (2021). Argument-based validation in testing and assessment. Sage. Google Scholar logo with link to Google Scholar
Chapelle, C. A., Enright, M. K., & Jamieson, J. (Eds.) (2008). Building a validity argument for the Test of English as a Foreign Language. Routledge.Google Scholar logo with link to Google Scholar
Coulmas, F. (1979). On the sociolinguistic relevance of routine formulae. Journal of Pragmatics, 31, 239–266. Google Scholar logo with link to Google Scholar
Davidson, D. E. (2010). Study abroad: When, how long, and with what results? New data from the Russian front. Foreign Language Annals, 43(1), 6–26. Google Scholar logo with link to Google Scholar
Di Silvio, F., Donovan, A., & Malone, M. E. (2014). The effect of study abroad homestay placements: Participant perspectives and oral proficiency gains. Foreign Language Annals, 47(1), 168–188. Google Scholar logo with link to Google Scholar
Ebel, R. L. (1964). The social consequences of educational testing. In Proceedings of the 1963 invitational conference on testing problems (pp. 130–143). Educational Testing Service.Google Scholar logo with link to Google Scholar
ETS. (2010). Linking TOEFL iBT scores to IELTS scores – A research report. ETS.Google Scholar logo with link to Google Scholar
Geeslin, K. L., & Schmidt, L. B. (2018). Study abroad and L2 learner attitudes. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 385–405). Routledge. Google Scholar logo with link to Google Scholar
Hulstijn, J. H. (2011). Language proficiency in native and nonnative speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly, 8(3), 229–249. Google Scholar logo with link to Google Scholar
Ikeda, N. (2017). Measuring L2 oral pragmatic abilities for use in social contexts: Development and validation of an assessment instrument for L2 pragmatics performance in university settings (Unpublished doctoral dissertation). The University of Melbourne, Australia.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.) (pp. 17–64). American Council on Education/Praeger Publishers.Google Scholar logo with link to Google Scholar
Matsumura, S. (2001). Learning the rules for offering advice: A quantitative approach to second language socialization. Language Learning, 51(4), 635–679. Google Scholar logo with link to Google Scholar
(2003). Modelling the relationships among interlanguage pragmatic development, L2 proficiency, and exposure to L2. Applied Linguistics, 24(4), 465–491. Google Scholar logo with link to Google Scholar
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed.) (pp. 13–103). American Council on Education & Macmillan.Google Scholar logo with link to Google Scholar
Myles, F., & Cordier, C. (2017). Formulaic sequence (FS) cannot be an umbrella term in SLA: Focusing on psycholinguistic FSs and their identification. Studies in Second Language Acquisition, 39(1), 3–28. Google Scholar logo with link to Google Scholar
Pawley, A., & Syder, F. H. (1983). Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. In J. C. Richards & R. W. Schmidt (Eds.), Language and communication (pp. 191–226). Longman.Google Scholar logo with link to Google Scholar
Powers, D. E., Roever, C., Huff, K., & Trapani, C. (2003). Validating LanguEdge Courseware Scores Against Faculty Ratings and Student Self-assessments. (ETS Research Report RR-03-11). Educational Testing Service. Google Scholar logo with link to Google Scholar
(2015). L2 pragmatic development in study abroad contexts. Peter Lang. Google Scholar logo with link to Google Scholar
(2019). Pragmatic development of Chinese during study abroad: A cross-sectional study of learner requests. Journal of Pragmatics, 1461, 137–149. Google Scholar logo with link to Google Scholar
Roever, C. (2005). Testing ESL pragmatics. Peter Lang. Google Scholar logo with link to Google Scholar
(2012). What learners get for free (and when): Learning of routine formulae in ESL and EFL environments. ELT Journal, 66(1), 10–21. Google Scholar logo with link to Google Scholar
Roever, C., Fraser, C., & Elder, C. (2014). Testing ESL sociopragmatics. Peter Lang. Google Scholar logo with link to Google Scholar
Roever, C., & Phakiti, A. (2018). Quantitative methods in language assessment and acquisition research. Routledge. Google Scholar logo with link to Google Scholar
Roever, C., Wang, S., & Brophy, S. (2014). Learner background factors and learning of second language pragmatics. International Review of Applied Linguistics, 52(4), 377–401. Google Scholar logo with link to Google Scholar
Sánchez-Hernández, A., & Alcón-Soler, E. (2019). Pragmatic gains in the study abroad context: Learners’ experiences and recognition of pragmatic routines. Journal of Pragmatics, 1461, 54–71. Google Scholar logo with link to Google Scholar
Schauer, G. A. (2006). Pragmatic awareness in ESL and EFL contexts: Contrast and development. Language Learning, 56(2), 269–318. Google Scholar logo with link to Google Scholar
Schauer, G. (2009). Interlanguage pragmatic development: The study abroad context. Bloomsbury Publishing.Google Scholar logo with link to Google Scholar
Schenker, T. (2018). Making short-term study abroad count – Effects on German language skills. Foreign Language Annals, 51(2), 411–429. Google Scholar logo with link to Google Scholar
Sell, F., Renkwitz, K., Sickinger, P., & Schneider, K. P. (2019). Measuring pragmatic competence on the functional and lexical level: The development of German high-school students’ requests during a stay abroad in Canada. Journal of Pragmatics, 1461, 106–120. Google Scholar logo with link to Google Scholar
Shively, R. L. (2013). Learning to be funny in Spanish during study abroad: L2 humor development. Modern Language Journal, 971, 930–946. Google Scholar logo with link to Google Scholar
(2015). Developing interactional competence during study abroad: Listener responses in L2 Spanish. System, 481, 86–98. Google Scholar logo with link to Google Scholar
Sireci, S., & Faulkner-Bond, M. (2014). Validity evidence based on test content. Psicothema, 26(1), 100–107. Google Scholar logo with link to Google Scholar
Sinclair, J. (1991). Corpus, concordance and collocation. Oxford University Press.Google Scholar logo with link to Google Scholar
Taguchi, N. (2014). Development of interactional competence in Japanese as a second language: Use of incomplete sentences as interactional resources. Modern Language Journal, 98(2), 518–535. Google Scholar logo with link to Google Scholar
Taguchi, N., Li, S., & Xiao, F. (2013). Production of formulaic expressions in L2 Chinese: A developmental investigation in a study abroad context. Chinese as a Second Language Research, 2(1), 23–58. Google Scholar logo with link to Google Scholar
Taguchi, N., Xiao, F., & Li, S. (2016). Effects of intercultural competence and social contact on speech act production in a Chinese study abroad context. Modern Language Journal, 1001, 775–796. Google Scholar logo with link to Google Scholar
Tannenbaum, R. J., & Baron, P. A. (2011). Mapping TOEFL® ITP scores onto the Common European Framework of Reference. ETS.Google Scholar logo with link to Google Scholar
Tare, M., Golonka, E., Lancaster, A. K., Bonilla, C., Doughty, C. J., Belnap, R. K., & Jackson, S. R. (2018). The role of cognitive aptitudes in a study abroad language-learning environment. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 406–420). Routledge. Google Scholar logo with link to Google Scholar
Tullock, B., & Ortega, L. (2017). Fluency and multilingualism in study abroad: Lessons from a scoping review. System, 711, 7–21. Google Scholar logo with link to Google Scholar
Xiao, F. (2015). Adult second language learners’ pragmatic development in the study-abroad context: A review. Frontiers: The Interdisciplinary Journal of Study Abroad, 251, 132–149. Google Scholar logo with link to Google Scholar
Xu, Y. (2019). Changes in interlanguage complexity during study abroad: A meta-analysis. System, 801, 199–211. Google Scholar logo with link to Google Scholar
Yang, J. (2016). CFL learners’ recognition and production of pragmatic routine formulae. Chinese as a Second Language. The Journal of the Chinese Language Teachers Association, USA, 51(1), 29–61. Google Scholar logo with link to Google Scholar
Youn, S. J. (2015). Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32(2), 199–225. Google Scholar logo with link to Google Scholar
Cited by (2)

Cited by two other publications

Roever, Carsten
2024. Assessment and L2 pragmatics. Applied Pragmatics 6:2 ► pp. 117 ff. DOI logo
Higuchi, Yuki, Makiko Nakamuro, Carsten Roever, Miyuki Sasaki & Tomoko Yashima
2023. Impact of studying abroad on language skill development: Regression discontinuity evidence from Japanese university students. Journal of the Japanese and International Economies 70 ► pp. 101284 ff. DOI logo

This list is based on CrossRef data as of 1 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue