Cover not available

Article published In: ITL - International Journal of Applied Linguistics
Vol. 173:1 (2022) ► pp.2–17

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Multimodal reading and second language learning

ORCiD logo with link | University College London
Published online: 1 February 2022
DOI logo with link
https://doi.org/10.1075/itl.21039.pel
Abstract
Most of the texts that second language learners engage with include both text (written and/or spoken) and images. The use of images accompanying texts is believed to support reading comprehension and facilitate learning. Despite their widespread use, very little is known about how the presentation of multiple input sources affects the attentional demands and the underlying cognitive processes involved. This paper provides a review of research on multimodal reading, with a focus on attentional processing. It first introduces the relevant theoretical frameworks and empirical evidence provided in support of the use of pictures in reading. It then reviews studies that have looked at the processing of text and pictures in first and second language contexts. Based on this review, main gaps in research and future research directions are identified. The discussion provided in this paper aims at advancing research on multimodal reading in a second language. Achieving a better understanding of the underlying cognitive processes in multimodal reading is crucial to inform pedagogical practices and to develop theoretical accounts of second language multimodal reading.
Article outline
  • Introduction
  • Multimodality in reading
  • Multimodal reading in second language learning
  • Cognitive processing of multimodal texts
  • Research gaps and future directions
    • Purposes of reading
    • Relationship between attentional allocation and learning
    • Image-Text relationship
  • Conclusion
  • References
References (70)
References
Abraham, P., & Farías, M. (2017). Reading with eyes wide open: Reflections on the impact of multimodal texts on second language reading. Íkala, Revista de Lenguaje y Cultura, 22(1), 57–70. Google Scholar logo with link to Google Scholar
Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Computers & Education, 1251, 413–428. Google Scholar logo with link to Google Scholar
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166–195. Google Scholar logo with link to Google Scholar
Bisson, M-J., W. Van Heuven, K. Conklin & R. Tunney. (2015). The role of verbal and pictorial information in multimodal incidental acquisition of foreign language vocabulary. Quarterly Journal of Experimental Psychology 681, 1306–1326. Google Scholar logo with link to Google Scholar
Boerma, I. E., Mol, S. E., & Jolles, J. (2016). Reading pictures for story comprehension requires mental imagery skills. Frontiers in Psychology, 7 (1630), 1–10. Google Scholar logo with link to Google Scholar
Buck, G. (2001). Assessing listening. New York, NY: Cambridge University Press. Google Scholar logo with link to Google Scholar
Center, Y., Freeman, L., Robertson, G., & Outhred, L. (1999). The effect of visual imagery training on the reading and listening comprehension of low listening comprehenders in year 2. Journal of Research in Reading 221, 241–256. Google Scholar logo with link to Google Scholar
Chang, Y., & Choi, S. (2014). Effects of seductive details evidenced by gaze duration. Neurobiology of Learning and Memory, 1091, 131–138. Google Scholar logo with link to Google Scholar
Chen, S. C., Hsiao, M. S., & She, H. C. (2015). The effects of static versus dynamic 3D representations on 10th grade students’ atomic orbital mental model construction: Evidence from eye movement behaviors. Computers in Human Behavior, 531, 169–180. Google Scholar logo with link to Google Scholar
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 31, 149–210. Google Scholar logo with link to Google Scholar
Eitel, A., & Scheiter, K. (2015). Picture or text first? Explaining sequence effects when learning with pictures and text. Educational Psychology Review, 271, 153–180. Google Scholar logo with link to Google Scholar
Elley, W. B. & F. Mangubhai. (1983). The impact of reading on second language learning. Reading Research Quarterly 19(1), 53–67. Google Scholar logo with link to Google Scholar
Evans, M. A. & J. Saint-Aubin. (2005). What children are looking at during shared storybook reading: Evidence from eye movement monitoring. Psychological Science 161, 913–920. Google Scholar logo with link to Google Scholar
Grabe, W. (2006). Areas of research that influence L2 reading instruction. In Usó-Juan, E. & Martínez-Flor, A. (Eds.), Current trends in the development and teaching of the four language skills (pp. 279–301). New York: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
(2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Godfroid, A., Boers, F., & Housen, A. (2013). An eye for words: Gauging the role of attention in incidental L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition, 35(3), 483–517. Google Scholar logo with link to Google Scholar
Guijarro, A. J. M., & Sanz, M. J. P. (2009). In interaction of image and verbal text in a picture book. A multimodal and systemic functional study. In Ventola, E., & Guijarro, A. J. M. (Eds.), The world told and the world shown (pp. 107–123). London: Palgrave Macmillan.Google Scholar logo with link to Google Scholar
Hannus, M., & Hyönä, J. (1999). Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology, 24(2), 95–123. Google Scholar logo with link to Google Scholar
Hill, D. (2013). Graded readers. ELT Journal 67(1), 85–125. Google Scholar logo with link to Google Scholar
Holsanova, J. (2014). Reception of multimodality: Applying eye tracking methodology in multimodal research. In C. Jewitt (Ed.), The Routledge Handbook of Multimodal Analysis (pp. 287–298). New York: Routledge.Google Scholar logo with link to Google Scholar
Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 181, 178–191. Google Scholar logo with link to Google Scholar
Justice, L. M., Skibbe, L., Canning, A., & Lankford, C. (2005). Pre-schoolers, print and storybooks: An observational study using eye movement analysis. Journal of Research in Reading, 281, 229–243. Google Scholar logo with link to Google Scholar
Kress, G. (2000). Multimodality: A social semiotic approach to communication. London: Routledge.Google Scholar logo with link to Google Scholar
Kress, G. & van Leeuwen, T. (2021). Reading images: The grammar of visual design. New York: Routledge.Google Scholar logo with link to Google Scholar
Lee, H., & Mayer, R. (2018). Fostering learning from instructional video in a second language. Applied Cognitive Psychology, 321, 648–654. Google Scholar logo with link to Google Scholar
Lee, M., & Révész, A. (2018). Promoting grammatical development through textually enhanced captions: An eye-tracking study. The Modern Language Journal, 1021, 557–577. Google Scholar logo with link to Google Scholar
Makransky, G., Terkildsen, T. S., & Mayer, R. E. (2019). Role of subjective and objective measures of cognitive processing during learning in explaining the spatial contiguity effect. Learning and Instruction, 611, 23–34. Google Scholar logo with link to Google Scholar
Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N. (2013a). An eye-tracking study of learning science text with concrete and abstract illustrations. Journal of Experimental Education, 811, 356–384. Google Scholar logo with link to Google Scholar
Mason, L., Tornatora, M. C., & Pluchino, P. (2013b). Do fourth graders integrate text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Computers and Education, 601, 95–109. Google Scholar logo with link to Google Scholar
(2015). Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: An eye movement study. Reading and Writing, 281, 851–872. Google Scholar logo with link to Google Scholar
Mayer, R. E. (2001). Multimedia learning. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
(2009). Multimedia learning (2nd ed.). Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
(2014a). Introduction to multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (pp. 1–24). Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Mayer, R. (2014b). Cognitive theory of multimedia learning. In R. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 43–71). Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Mayer, R. E., & Anderson, R. B. (1991). Animations need narrations: An experimental test of a dual-coding hypothesis. Journal of Educational Psychology, 83(4), 484–490. Google Scholar logo with link to Google Scholar
Mayer, R. E., Howarth, J. T., Kaplan, M., & Hanna, S. (2018). Applying the segmenting principle to online geography slideshow lessons. Educational Technology Research and Development, 66(3), 563–577. Google Scholar logo with link to Google Scholar
Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90(2), 312–320. Google Scholar logo with link to Google Scholar
Mills, K., & Unsworth, L. (2017). Multimodal literacy. Oxford research encyclopedia of education. Retrieved 22 Jul. 2021, from: [URL]
Montero Perez, M. (2020). Multimodal input in SLA research. Studies in Second Language Acquisition, 42(3), 653–663. Google Scholar logo with link to Google Scholar
Montero Perez, M., Van Den Noortgate, W., & Desmet, P. (2013). Captioned video for L2 listening and vocabulary learning: A meta-analysis. System, 411, 720–739. Google Scholar logo with link to Google Scholar
Montero Perez, M., Peters, E., & Desmet, P. (2015). Enhancing vocabulary learning through captioned video: an eye-tracking study. The Modern Language Journal, 99(2), 308–28. Google Scholar logo with link to Google Scholar
Moreno, R., & Mayer, R. E. (1999). Multimedia-supported metaphors for meaning making in mathematics. Cognition and Instruction, 17(3), 215–248. Google Scholar logo with link to Google Scholar
(2002a). Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Educational Psychology, 94(3), 598–610. Google Scholar logo with link to Google Scholar
(2002b). Verbal redundancy in multimedia learning: When reading helps listening. Journal of Educational Psychology, 94(1), 156–163. Google Scholar logo with link to Google Scholar
Omaggio, A. C. (1979). Pictures and second language comprehension: Do they help? Foreign Language Annals, 121, 107–116. Google Scholar logo with link to Google Scholar
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
(2006). Mind and its evolution: A dual coding approach. Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Pellicer-Sánchez, A. (2016). Incidental L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second Language Acquisition, 381, 97–130. Google Scholar logo with link to Google Scholar
Pellicer-Sánchez, A., Conklin, K., Rodgers, M. P., & Parente, F. (2021), The Effect of Auditory Input on Multimodal Reading Comprehension: An Examination of Adult Readers’ Eye Movements. The Modern Language Journal, 105(4), 936–956: Google Scholar logo with link to Google Scholar
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Llanes, A., & Serrano, R. (2018). L2 reading and reading-while-listening in multimodal learning conditions: An eye-tracking study. ELT Research Papers, 18(01), 1–28. London: British Council.Google Scholar logo with link to Google Scholar
Pellicer-Sánchez, A., Tragant, E., Conklin, K., Rodgers, M., Serrano, R., & Llanes, A. (2020). Young learners’ processing of multimodal input and its impact on reading comprehension: An eye-tracking study. Studies in Second Language Acquisition, 42(3), 577–598. Google Scholar logo with link to Google Scholar
Peters, E., & Muñoz, C. (2020). Introduction to the special issue: Language learning from multimodal input. Studies in Second Language Acquisition, 42(3), 489–497. Google Scholar logo with link to Google Scholar
Plass, J. L., Chun, D. M., Mayer, R. E., & Leutner, D. (1998). Supporting visual and verbal learning preferences in a second language multimedia learning environment. Journal of Educational Psychology, 901, 25–36. Google Scholar logo with link to Google Scholar
Puimège, E., Montero Perez, M., & Peters, E. (2021). Promoting L2 acquisition of multiword units through textually enhanced audiovisual input: An eye-tracking study. Second Language Research. Online First. Google Scholar logo with link to Google Scholar
Rodgers, M. P. H., & Webb, S. (2017). The effects of captions on EFL learners’ comprehension of English-language television programs. Calico Journal, 341, 20–38. Google Scholar logo with link to Google Scholar
Rodgers, M. P. H. & Webb, S. (2019). Incidental vocabulary learning through viewing television. ITL-International Journal of Applied Linguistics, 171(2), 191–220. Google Scholar logo with link to Google Scholar
Roy-Charland, A., Saint-Aubin, J., & Evans, M. A. (2007). Eye movements in shared book reading with children from kindergarten to grade 4. Reading and Writing, 201, 909–931. Google Scholar logo with link to Google Scholar
Schmidt-Weigand, F. (2011). Does animation amplify the modality effect–or is there any modality effect at all? Zeitschrift für Padagogische Psychologie, 25(4), 245–256. Google Scholar logo with link to Google Scholar
Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). A closer look at split visual attention in system-and self-paced instruction in multimedia learning. Learning and Instruction, 20(2), 100–110. Google Scholar logo with link to Google Scholar
Schnotz, W., & Baadte, C. (2008). Domain learning versus language learning with multimedia. In M. Farías & K. Obilinovic (Eds.), Aprendizaje multimodal/Multimodal learning (pp. 21–49). Santiago de Chile: Publifahu USACH.Google Scholar logo with link to Google Scholar
Seburn, T. (2017). Learner-sourced visuals for deeper text engagement and conceptual comprehension. In K. Donaghy & D. Xerri (Eds.), The image in English language teaching (pp. 79–88). Malta: Gutenberg Press.Google Scholar logo with link to Google Scholar
Serrano, R., & Pellicer-Sánchez, A. (2019). Young L2 learners’ online processing of information in a graded reader during reading-only and reading-while-listening conditions: A study of eye movements. Applied Linguistics Review. Advance online publication. Google Scholar logo with link to Google Scholar
Takacs, Z. K., & Bus, A. G. (2016). Benefits of motion in animated storybooks for children’s visual attention and story comprehension. An eye-tracking study. Frontiers in Psychology, 71, 1–12. Google Scholar logo with link to Google Scholar
Tang, G. (1992). The effect of graphic representation of knowledge structures on ESL reading comprehension. Studies in Second Language Acquisition, 141, 177–195. Google Scholar logo with link to Google Scholar
Tragant, E., & Pellicer-Sánchez, A. (2019). Young learners’ engagement with multimodal exposure: An eye tracking study. System, 801, 212–223. Google Scholar logo with link to Google Scholar
Unsworth, L. (2006). Towards a metalanguage for multiliteracies education: Describing the meaning – making resources of language-image interaction. English Teaching, 5(1), 55–76.Google Scholar logo with link to Google Scholar
(2014). Multimodal reading comprehension: Curriculum expectations and large-scale literacy testing practices. Pedagogies: An International Journal, 9(1), 26–44. Google Scholar logo with link to Google Scholar
Winke, P., Gass, S., & Sydorenko, T. (2010). The effects of captioning videos used for foreign language listening activities. Language Learning & Technology, 141, 65–86.
CrossRef DOI logo with hyperlink to permanent DOI
Google Scholar logo with link to Google Scholar
Wright, A. (2010). Pictures for language learning. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Cited by (13)

Cited by 13 other publications

Carcamo, Benjamin & Bernardo Pino
2025. Developing EFL students’ multimodal literacy with the use of infographics. Asian-Pacific Journal of Second and Foreign Language Education 10:1 DOI logo
Mahshanian, Amir, Mohammadtaghi Shahnazari & Ahmad Moinzadeh
2025. Investigating the Relationship Between Working Memory, Semantic Encoding, and L2 Reading Ability Across Proficiency Levels. Journal of College Reading and Learning 55:1 ► pp. 63 ff. DOI logo
Pu, Pu, Daniel Yu‐Sheng Chang & Shuang Wang
2025. Young EFL Learners’ Incidental Collocation Learning Through Different Multimodal Inputs: A Mixed‐Methods Study. International Journal of Applied Linguistics DOI logo
Ramazanova, M.A., M.S. Kulakhmetova & Zh.A. Tusselbayeva
2025. SOME ASPECTS OF MULTIMODAL APPROACH IN SECOND FOREIGN LANGUAGE TEACHING. Журнал серии «Педагогические науки» 77:2 DOI logo
Avello, Daniela & Carmen Muñoz
2024. The development of L1 and L2 reading skills from captioned video viewing in primary school EFL learners. In Audiovisual input and second language learning [Language Learning & Language Teaching, 61], ► pp. 25 ff. DOI logo
Avello, Daniela & Carmen Muñoz
2025. Primary school learners benefit from captioned video viewing. In Early Language Education in Instructed Contexts [Language Learning & Language Teaching, 62], ► pp. 43 ff. DOI logo
Avello, Daniela & Carmen Muñoz
2025. Vocabulary Learning from Captioned Video Viewing in Primary School Learners: The Influence of Word- and Context-Related Factors. In Foreign Language Learning from Audiovisual Input [Educational Linguistics, 66], ► pp. 17 ff. DOI logo
Vanbuel, Marieke & Bart Deygers
2024. Assessing Receptive Skills in an Adult L2 Learner Population with Diverse Educational Backgrounds. Language Assessment Quarterly 21:4-5 ► pp. 379 ff. DOI logo
Cárdenas-Claros, Mónica Stella, Tetyana Sydorenko, Elizabeth Huntley & Maribel Montero Perez
2023. Teachers’ voices on multimodal input for second or foreign language learning. Language Teaching Research DOI logo
Marzban, Sheida, Habil Gyöngyi Fábián & Béla Weiss
2023. 2023 Symposium on Eye Tracking Research and Applications, ► pp. 1 ff. DOI logo
Reynolds, Barry Lee
2023. Introduction: A Scoping Review of Second Language Vocabulary Learning in the Wild. In Vocabulary Learning in the Wild, ► pp. 1 ff. DOI logo
Reynolds, Barry Lee, Xiaowen (Serina) Xie & Quy Huynh Phu Pham
2023. The potentials for incidental vocabulary acquisition from listening to computer science academic lectures: a higher education corpus-based case study from Macau. Frontiers in Psychology 14 DOI logo
Reynolds, Barry Lee, Yanwen Cui, Chian-Wen Kao & Nathan Thomas
2022. Vocabulary Acquisition through Viewing Captioned and Subtitled Video: A Scoping Review and Meta-Analysis. Systems 10:5 ► pp. 133 ff. DOI logo

This list is based on CrossRef data as of 12 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue