Cover not available

In:Technology and Instructed Second Language Acquisition: Connecting research and pedagogy
Edited by Shawn Loewen, Frederick J. Poole, Hyun-Bin Hwang and Matthew D. Coss
[Language Learning & Language Teaching 63] 2025
► pp. 139–162

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Chapter 6
The impact of multimodal input tools in instructed second language acquisition (and beyond)

 | Ghent University
Published online: 27 October 2025
DOI logo with link
https://doi.org/10.1075/lllt.63.06mon
Abstract
Multimodal input tools combine various input modes, such as imagery, text, and auditory information. Examples of such tools are reading-while-listening activities, texts with multimodal annotations, and videos with on-screen text, such as L1 subtitles or captions. The effectiveness of multimodal input tools has been studied for various aspects of L2 learning (e.g., vocabulary, input comprehension). Research has also addressed L2 learners’ use and processing of multimodal input. These questions have been examined from various theoretical perspectives, including usage-based approaches, cognitively-oriented SLA theories on attention and learning, and the cognitive theory of multimedia learning. This chapter examines recent findings on multimodal input tools, considers their implications for ISLA and discusses limitations that may affect the generalizability of current research.
Article outline
  • Introduction
  • Theories and research
    • Theoretical perspectives on multimodal input tools
    • Multimodal input in ISLA
    • Exemplary multimodal input and instructed SLA study
    • Generalizability and representativeness
  • Pedagogical applications and considerations
    • Make written texts multimodal through multimodal annotations
    • Use reading-while-listening activities to promote L2 reading and vocabulary learning
    • Increase input comprehension by combining written and spoken input modes in video-based types of multimodal input
    • Foster vocabulary learning through audiovisual input
    • Promote word learning through a combination of audiovisual input and written text
    • Use sequenced viewing activities to promote various aspects of L2 learning
    • Reflect on L2 learning gains that can be expected from video-based types of multimodal input
    • Consider lexical characteristics when selecting multimodal input
    • Promote learners’ out-of-class exposure to multimodal input
  • Pedagogical scenario for reflection
  • Extension resources
    • Reflection and discussion questions
    • Topics for (action) research projects
  • Generative AI use acknowledgement
  • Acknowledgements
  • Notes
  • References
References (70)
References
Andringa, S., & Godfroid, A. (2020). Sampling bias and the problem of generalizability in applied linguistics. Annual Review of Applied Linguistics, 40, 134–142. Google Scholar logo with link to Google Scholar
Bednarek, M. (2018). Language and television series. A linguistic approach to TV dialogue. Cambridge University Press. Google Scholar logo with link to Google Scholar
Birulés-Muntané, J., & Soto-Faraco, S. (2016). Watching subtitled films can help learning foreign languages. PLoS One, 11(6), 1–10. Google Scholar logo with link to Google Scholar
Bisson, M. J., van Heuven, W. J. B., Conklin, K., & Tunney, R. J. (2014). Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35(2), 399–418. Google Scholar logo with link to Google Scholar
Boers, F. (2022). Glossing and vocabulary learning. Language Teaching, 55(1), 1–23. Google Scholar logo with link to Google Scholar
Boers, F., Warren, P., Grimshaw, G., & Siyanova-Chanturia, A. (2017). On the benefits of multimodal annotations for vocabulary uptake from reading. Computer Assisted Language Learning, 30(7), 709–725. Google Scholar logo with link to Google Scholar
Chang, A. C. S., & Millett, S. (2015). Improving reading rates and comprehension through audio-assisted extensive reading for beginner learners. System, 52, 91–102. Google Scholar logo with link to Google Scholar
Cintrón-Valentín, M. C., & García-Amaya, L. (2021). Investigating textual enhancement and captions in L2 grammar and vocabulary. Studies in Second Language Acquisition, 43(5), 1068–1093. Google Scholar logo with link to Google Scholar
Conklin, K., Alotaibi, S., Pellicer-Sánchez, A., & Vilkaitė-Lozdienė, L. (2020). What eye-tracking tells us about reading-only and reading-while-listening in a first and second language. Second Language Research, 36(3), 257–276. Google Scholar logo with link to Google Scholar
Crosthwaite, P., & Schweinberger, M. (2025). Corpora and instructed second language acquisition: A useful methodological synergy? In S. Loewen, F. Poole, H. Hwang, & M. D. Coss (Eds.), Technology and instructed second language acquisition: Connecting research and pedagogy (pp. 115–138). John Benjamins. (this volume)Google Scholar logo with link to Google Scholar
De Wilde, V., Brysbaert, M., & Eyckmans, J. (2020). Learning English through out-of-school exposure: How do word-related variables and proficiency influence receptive vocabulary learning? Language Learning, 70(2), 349–381. Google Scholar logo with link to Google Scholar
Duong, P. T., Montero Perez, M., Desmet, P., & Peters, E. (2021). Learning vocabulary in spoken input- and output-based tasks. TASK, Journal on Task-based Language Teaching and Learning, 1(1), 100–126. Google Scholar logo with link to Google Scholar
Durbahn, M., Rodgers, M., & Peters, E. (2020). The relationship between vocabulary and viewing comprehension. System, 88, 102–166. Google Scholar logo with link to Google Scholar
Fievez, I., Montero Perez, M., Cornillie, F., & Desmet, P. (2023). Promoting incidental vocabulary learning through watching a French Netflix series with glossed captions. Computer Assisted Language Learning 36(1), 26–51. Google Scholar logo with link to Google Scholar
Galimberti, V., Mora, J. C., & Gilabert, R. (2023). Audio-synchronized textual enhancement in foreign language pronunciation learning from videos. System, 116, 103078. Google Scholar logo with link to Google Scholar
Gass, S., Winke, S., Isbell, P., & Ahn, D. R. (2019). How captions help people learn languages: A working-memory, eye-tracking study. Language Learning & Technology 23(2), 84–104. Google Scholar logo with link to Google Scholar
Hui, B. (2024). Scaffolding comprehension with reading while listening and the role of reading speed and text complexity. Modern Language Journal, 108(1), 183–200. Google Scholar logo with link to Google Scholar
Kurokawa, S., Hein, A. M., & Uchihara, T. (2025). Incidental vocabulary acquisition through captioned viewing: A meta-analysis. Language Learning. Advance online publication. Google Scholar logo with link to Google Scholar
Lai, C. (2025). Connecting instructed second language learning with informal language learning with technology. In S. Loewen, F. Poole, H. Hwang, & M. D. Coss (Eds.), Technology and instructed second language acquisition: Connecting research and pedagogy (pp. 92–114). John Benjamins. (this volume)Google Scholar logo with link to Google Scholar
Laufer, B., & Ravenhorst-Kalovski, G. C. (2010). Lexical threshold revisited: Lexical text coverage, learners’ vocabulary size and reading comprehension. Reading in a Foreign Language, 22(1), 15.Google Scholar logo with link to Google Scholar
Lee, M., & Révész, A. (2020). Promoting grammatical development through captions and textual enhancement in multimodal input-based tasks. Studies in Second Language Acquisition, 42(3), 1–27. Google Scholar logo with link to Google Scholar
Lin, P. (2022). Developing an intelligent tool for computer-assisted formulaic language learning from YouTube videos. ReCALL, 34(2), 185–200. Google Scholar logo with link to Google Scholar
Lindgren, E., & Muñoz, C. (2013). The influence of exposure, parents, and linguistic distance on young European learners’ foreign language comprehension. International Journal of Multilingualism, 10(1), 105–129. Google Scholar logo with link to Google Scholar
Loewen, S. (2011). Focus on form. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (Vol. 2, pp. 576–592). Routledge.Google Scholar logo with link to Google Scholar
Majuddin, E., Siyanova-Chanturia, A., & Boers, F. (2021). Incidental acquisition of multiword expressions through audiovisual materials. Studies in Second Language Acquisition, 43(5), 985–1008. Google Scholar logo with link to Google Scholar
Malone, J. (2018). Incidental vocabulary learning in SLA: Effects of frequency, aural enhancement, and working memory. Studies in Second Language Acquisition, 40, 651–675. Google Scholar logo with link to Google Scholar
Mayer, R. E. (2001). Multimedia learning. Cambridge University Press. Google Scholar logo with link to Google Scholar
Mayer, R. E., Fiorella, L., & Stull, A. (2020). Five ways to increase the effectiveness of instructional video. Educational Technology Research and Development, 68, 837–852. Google Scholar logo with link to Google Scholar
McManus, K. (2024). Introducing usage in SLA. In K. McManus, Usage in Second Language Acquisition (pp. 1–25). Routledge. Google Scholar logo with link to Google Scholar
Mitterer, H., & McQueen, J. M. (2009). Foreign subtitles help but native-language subtitles harm foreign speech perception. PLoS One, 4(11), 4–8. Google Scholar logo with link to Google Scholar
Montero Perez, M. (2020). Incidental vocabulary learning through viewing video. The role of vocabulary knowledge and working memory. Studies in Second Language Acquisition, 42(4), 749–773. Google Scholar logo with link to Google Scholar
Multimodal input and language learning: a research-practice gap? Ghent university SLA research seminar, March 2025.Google Scholar logo with link to Google Scholar
Montero Perez, M. & Peters, E. (in preparation). Captioned video. A guide for research and practice.
Montero Perez, M., Peters, E., & Desmet, P. (2014). Is less more? Effectiveness and perceived usefulness of keyword and full captioned video for L2 listening comprehension. ReCALL, 26(1), 21–43. Google Scholar logo with link to Google Scholar
(2018). Vocabulary learning through viewing video: The effect of two enhancement techniques. Computer Assisted Language Learning, 31(1–2), 1–26. Google Scholar logo with link to Google Scholar
Muñoz, C. (2017). The role of age and proficiency in subtitle reading. An eye-tracking study. System, 67, 77–86. Google Scholar logo with link to Google Scholar
Nation, P. (2007). The four strands. Innovation in Language Learning and Teaching, 1(1), 1–13. Google Scholar logo with link to Google Scholar
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.Google Scholar logo with link to Google Scholar
Pattemore, A., Cabrera Fernández, B., Lopez de Artola, M., & Michel, M. (2024). Maximising the potential of plurilingual subtitled audiovisual input: Learning L3 words and multiword units with L2 audio. Innovation in Language Learning and Teaching, 1–16. Google Scholar logo with link to Google Scholar
Pattemore, A., & Muñoz, C. (2020). Learning L2 constructions from captioned audiovisual exposure: The effect of learner-related factors. System, 93, 102303. Google Scholar logo with link to Google Scholar
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Serrano, R., & Llanes, Á. (2020). Young learners’ processing of multimodal input and its impact on reading comprehension: An eye-tracking study. Studies in Second Language Acquisition, 42(3), 577–598. Google Scholar logo with link to Google Scholar
(2019). The effect of imagery and on-screen text on foreign language vocabulary learning from audio-visual input. TESOL Quarterly, 53(4), 1008–1032. Google Scholar logo with link to Google Scholar
Peters, E., Heynen, E., & Puimège, E. (2016). Learning vocabulary through audio-visual input: The differential effect of L1 subtitles and captions. System, 63, 134–148. Google Scholar logo with link to Google Scholar
Peters, E., & Webb, S. (2018). Incidental vocabulary acquisition through viewing L2 television and factors that affect learning. Studies in Second Language Acquisition, 40(3), 551–577. Google Scholar logo with link to Google Scholar
Puimège, E., & Peters, E. (2020). Learning formulaic sequences through viewing L2 television and factors that affect learning. Studies in Second Language Acquisition, 42(3), 525–549. Google Scholar logo with link to Google Scholar
Puimège, E., Montero Perez, M., & Peters, E. (2023). Promoting L2 acquisition of multiword units through textually enhanced audiovisual input: An eye-tracking study. Second Language Research, 39(2), 471–492. Google Scholar logo with link to Google Scholar
Pujadas, G., & Muñoz, C. (2019). Extensive viewing of captioned and subtitled TV series: A study of L2 vocabulary learning by adolescents. The Language Learning Journal, 47(4), 479–496. Google Scholar logo with link to Google Scholar
(2020). Examining adolescent EFL learners’ TV viewing comprehension through captions and subtitles. Studies in Second Language Acquisition, 42(3), 1–25. Google Scholar logo with link to Google Scholar
Quinn, S. D., & Zaykovskaya, I. (2025). Instructed SLA and instructional design principles: Making the best use of authoring tools in curriculum development. In S. Loewen, F. Poole, H. Hwang, & M. D. Coss (Eds.), Technology and instructed second language acquisition: Connecting research and pedagogy (pp. 68–91). John Benjamins. (this volume)Google Scholar logo with link to Google Scholar
Ramezanali, N., & Faez, F. (2019). Vocabulary learning and retention through multimedia glossing. Language Learning & Technology, 23(2), 105–124. Google Scholar logo with link to Google Scholar
Ramezanali, N., Uchihara, T., & Faez, F. (2021). Efficacy of multimodal glossing on L2 vocabulary learning: A meta-analysis. TESOL Quarterly, 55(1), 105–133. Google Scholar logo with link to Google Scholar
Rodgers, M. (2016). Extensive listening and viewing: The benefits of audiobooks and television. The European Journal of Applied Linguistics and TEFL, 5(2), 43–57.Google Scholar logo with link to Google Scholar
(2018). The images in television programs and the potential for learning unknown words. ITL — International Journal of Applied Linguistics, 169(1), 191–211. Google Scholar logo with link to Google Scholar
Saito, K., Uchihara, T., Takizawa, K., & Suzukida, Y. (2023). Individual differences in L2 listening proficiency revisited: Roles of form, meaning, and use aspects of phonological vocabulary knowledge. Studies in Second Language Acquisition, 1–27. Google Scholar logo with link to Google Scholar
Schmidt, R. W. (2001). Attention. In P. Robinson (Ed.), Cognition and second language instruction (pp. 3–32). Cambridge University Press. Google Scholar logo with link to Google Scholar
Sydorenko, T. (2010). Modality of input and vocabulary acquisition. Language Learning & Technology, 14(2), 50–73. Google Scholar logo with link to Google Scholar
Sydorenko, T., Cárdenas-Claros, M. S., Huntley, E., & Montero Perez, M. (2024). Audiovisual input in language learning: Teachers’ perspectives. Language Learning & Technology, 28(1).Google Scholar logo with link to Google Scholar
Tarone, E. (2010). Second language acquisition by low-literate learners: An under-studied population. Language Teaching, 43(1), 75–83. Google Scholar logo with link to Google Scholar
Vu, D. V., & Peters, E. (2022). Incidental learning of collocations from meaningful input: A longitudinal study into three reading modes and factors that affect learning. Studies in Second Language Acquisition, 44(3), 685–707. Google Scholar logo with link to Google Scholar
Wang, A. & Pellicer-Sánchez, A. (2023). Examining the effectiveness of bilingual subtitles for comprehension: An eye-tracking study. Studies in Second Language Acquisition, 45, 882–905. Google Scholar logo with link to Google Scholar
Warren, P., Boers, F., Grimshaw, G., & Siyanova-Chanturia, A. (2018). The effect of gloss type on learners’ intake of new words during reading: Evidence from eye-tracking. Studies in Second Language Acquisition, 40(4), 883–906. Google Scholar logo with link to Google Scholar
Webb, S., & Rodgers, M. (2009). Vocabulary demands of television programs. Language Learning, 59(2), 335–366. Google Scholar logo with link to Google Scholar
Webb, S. (2010). Pre-learning low-frequency vocabulary in second language television programmes. Language Teaching Research, 14(4), 501–515. Google Scholar logo with link to Google Scholar
(2015). Extensive viewing: Language learning through watching television. In D. Nunan & J. C. Richards (Eds.), Language learning beyond the classroom (pp. 159–168). Routledge.Google Scholar logo with link to Google Scholar
Wi, I., & Boers, F. (2023). Sequential use of L1 and L2 captions: Exploring the benefits for vocabulary acquisition. TESOL Quarterly. Google Scholar logo with link to Google Scholar
Winke, P., Gass, S., & Sydorenko, T. (2010). The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 14(1), 65–86. Google Scholar logo with link to Google Scholar
Winke, P., Sydorenko, T., & Gass, S. (2013). Factors influencing the use of captions by foreign language learners: An Eyetracking study. Modern Language Journal, 97(1), 254–275. Google Scholar logo with link to Google Scholar
Wisniewska, N., & Mora, J.#x202f;C. (2020). Can captioned video benefit second language pronunciation? Studies in Second Language Acquisition, 42(3), 1–26. Google Scholar logo with link to Google Scholar
Cited by (4)

Cited by four other publications

Buendgens-Kosten, Jules & Carolyn Blume
2025. Of barriers and bias. In Technology and Instructed Second Language Acquisition [Language Learning & Language Teaching, 63], ► pp. 45 ff. DOI logo
Poole, Frederick J. & Mireia Toda Cosi
2025. Generative AI in instructed SLA. In Technology and Instructed Second Language Acquisition [Language Learning & Language Teaching, 63], ► pp. 163 ff. DOI logo
Quinn, Shannon Donnally & Irina Zaykovskaya
2025. Instructed SLA and instructional design principles. In Technology and Instructed Second Language Acquisition [Language Learning & Language Teaching, 63], ► pp. 68 ff. DOI logo
Ziegler, Nicole
2025. Text/video chat and other online collaboration tools. In Technology and Instructed Second Language Acquisition [Language Learning & Language Teaching, 63], ► pp. 271 ff. DOI logo

This list is based on CrossRef data as of 29 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue