Cover not available

Article published In: Audiovisual Translation: Theoretical and methodological challenges
Edited by Yves Gambier and Sara Ramos Pinto
[Target 28:2] 2016
► pp. 276–287

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Psycholinguistics and audiovisual translation

 | Macquarie University (Australia) / North-West University (South Africa)
Published online: 9 August 2016
DOI logo with link
https://doi.org/10.1075/target.28.2.08kru
Abstract
Psycholinguistic investigations of translated audiovisual products have been conducted since at least the 1980s. These mainly concerned the role of subtitles in the processing of language in the context of language acquisition, literacy, and education. This article provides an overview of some of the most productive lines of research from a psycholinguistic angle in audiovisual translation (AVT), focussing on studies that investigated the positive effects of subtitles on language performance, but also on a growing body of behavioural research on the cognitive processing of the language of subtitles. The article evaluates a number of methodologies in some of the most prominent studies on the processing of subtitles, primarily making use of eye tracking, and then provides some thoughts on future directions in psycholinguistic studies on the processing of the language of AVT.
Article outline
  • 1.Introduction
  • 2.Performance and reception studies
  • 3.Physiological studies
  • 4.Eye tracking
  • 5.Conclusion: future psycholinguistic methodologies for AVT
  • References
References (27)
References
Bird, Stephen A., and John Williams. 2002. “The Effect of Bimodal Input on Implicit and Explicit Memory: An Investigation into the Benefits of Within-Language Subtitling.” Applied Psycholinguistics 23 (4): 509–533. Google Scholar logo with link to Google Scholar
Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney. 2012. “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 351: 399–418. Google Scholar logo with link to Google Scholar
Danan, Martine. 2004. “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77. Google Scholar logo with link to Google Scholar
Diao, Yali, Paul Chandler, and John Sweller. 2007. “The Effect of Written Text on Comprehension of Spoken English as a Foreign Language.” The American Journal of Psychology 120 (2): 237–261.Google Scholar logo with link to Google Scholar
d’Ydewalle, Géry, and Wim De Bruycker. 2007. “Eye Movements of Children and Adults While Reading Television Subtitles.” European Psychologist 121: 196–205. Google Scholar logo with link to Google Scholar
d’Ydewalle, Géry, and Ingrid Gielen. 1992. “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, ed. by Keith Rayner, 415–427. New York: Springer-Verlag. Google Scholar logo with link to Google Scholar
d’Ydewalle, Géry, Caroline Praet, Karl Verfaillie, and Johan van Rensbergen. 1991. “Watching Subtitled Television: Automatic Reading Behavior.” Communication Research 181: 650–665. Google Scholar logo with link to Google Scholar
Garza, Thomas J.. 1991. “Evaluating the Use of Captioned Video Materials in Advanced Foreign Language Learning.” Foreign Language Annuals 24 (3): 239–258. Google Scholar logo with link to Google Scholar
Ghia, Elisa. 2012. “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, ed. by Elisa Perego, 155–182. Roma: Aracne Editrice.Google Scholar logo with link to Google Scholar
Just, Marcel Adam, and Patricia A. Carpenter. 1980. “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354. Google Scholar logo with link to Google Scholar
Keating, Gregory D. 2014. “Eye Tracking with Text.” In Research Methods in Second Language Psycholinguistics, ed. by Jill Jegerski, and Bill VanPatten, 69–92. New York: Routledge.Google Scholar logo with link to Google Scholar
Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz. 2013. “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis. 2013. “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47 (1): 29–53.Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis, and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120. Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013. “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of the 2013 Conference on Eye Tracking South Africa, 62–66. New York: ACM. Google Scholar logo with link to Google Scholar
Markham, Peter L. 1993. “Captioned Television Videotapes: Effects of Visual Support on Second Language Comprehension.” Journal of Educational Technology Systems 211: 183–191. Google Scholar logo with link to Google Scholar
Markham Peter, L. 1999. “Captioned Video-Tapes and Second Language Listening Word Recognition.” Foreign Language Annals 32 (3): 321–328. Google Scholar logo with link to Google Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–272. Google Scholar logo with link to Google Scholar
Pulvermüller, Friedemann. 2009. “Word Processing in the Brain as Revealed by Neuro­physiological Imaging.” In The Oxford Handbook of Psycholinguistics, ed. by M. Gareth Gaskell, 119–139. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco. 2013. “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21 (1): 5–31. Google Scholar logo with link to Google Scholar
Rayner, Keith. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 1241: 372–422. Google Scholar logo with link to Google Scholar
Specker, Elizabeth A. 2008. L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. PhD diss. University of Arizona.Google Scholar logo with link to Google Scholar
Vanderplank, Robert. 1988. “The Value of Teletext Sub-titles in Language Learning.” ELT Journal 42 (4): 272–281. Google Scholar logo with link to Google Scholar
. 1990. “Paying Attention to the Words: Practical and Theoretical Problems in Watching Television Programmes with Uni-lingual (CEEFAX) Sub-titles.” System 18 (2): 221–234. Google Scholar logo with link to Google Scholar
. 2013. “‘Effects of’ and ‘Effects with’ Captions: How Exactly Does Watching a TV Programme with Same-Language Subtitles Make a Difference to Language Learners?” Language Teaching 1–16.Google Scholar logo with link to Google Scholar
VanPatten, Bill. 2014. “The Psycholinguistics of SLA.” In Research Methods in Second Language Psycholinguistics (Second Language Acquisition Research Series), ed. by Jill Jegerski, and Bill VanPatten, 1–19. New York: Routledge.Google Scholar logo with link to Google Scholar
Winke, Paula, Susan Gass, and Tetyana Syderenko. 2013. “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1): 254–275. Google Scholar logo with link to Google Scholar
Cited by (17)

Cited by 17 other publications

Brescia-Zapata, Marta, Krzysztof Krejtz, Andrew T. Duchowski, Chris J. Hughes & Pilar Orero
2025. Subtitles in VR 360° video. Results from an eye-tracking experiment. Perspectives 33:2 ► pp. 357 ff. DOI logo
El Abed, Maya & Mary Yazbeck
2025. Subtitling versus dubbing in animated films: cognitive and evaluative reception in young children. Cogent Arts & Humanities 12:1 DOI logo
Carl, Michael & Isabel Lacruz
2024. Experimental Methods in Translation Studies. In Reference Module in Social Sciences, DOI logo
Fouz-González, Jonás & Joan C. Mora
2024. Exploring the potential of textually‑enhanced captioned video to direct learners’ attention to challenging sound contrasts. Journal of Second Language Pronunciation 10:3 ► pp. 309 ff. DOI logo
Ragni, Valentina
2023. Reverse subtitles in foreign language learning. Translation and Translanguaging in Multilingual Contexts 9:2 ► pp. 261 ff. DOI logo
Xiao, Kairong & Wanyu Li
2023. In the eye of the beholder. In Translation in Transition [American Translators Association Scholarly Monograph Series, XX], ► pp. 104 ff. DOI logo
Hermosa-Ramírez, Irene
2022. Physiological instruments meet mixed methods in Media Accessibility. Translation Spaces 11:1 ► pp. 38 ff. DOI logo
Kruger, Jan-Louis & Sixin Liao
2022. Establishing a theoretical framework for AVT research. Translation Spaces 11:1 ► pp. 12 ff. DOI logo
Liao, Sixin, Lili Yu, Jan-Louis Kruger & Erik D. Reichle
2022. The impact of audio on the reading of intralingual versus interlingual subtitles: Evidence from eye movements. Applied Psycholinguistics 43:1 ► pp. 237 ff. DOI logo
Guillot, Marie-Noëlle
2020. Ocean’s Eleven Scene 12: the sample as methodological cogitation – rationale and data. Perspectives 28:6 ► pp. 816 ff. DOI logo
Pedersen, Jan
2020. Audiovisual Translation Norms and Guidelines. In The Palgrave Handbook of Audiovisual Translation and Media Accessibility [Palgrave Studies in Translating and Interpreting, ], ► pp. 417 ff. DOI logo
Walker, Callum
2019. A cognitive perspective on equivalent effect: using eye tracking to measure equivalence in source text and target text cognitive effects on readers. Perspectives 27:1 ► pp. 124 ff. DOI logo
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff. DOI logo
Federici, Federico M. & Callum Walker
2018. A mapping exercise. In Eye tracking and multidisciplinary studies on translation [Benjamins Translation Library, 143], ► pp. 11 ff. DOI logo
Kruger, Jan-Louis & Stephen Doherty
2018. Triangulation of online and offline measures of processing and reception in AVT. In Reception Studies and Audiovisual Translation [Benjamins Translation Library, 141], ► pp. 91 ff. DOI logo
Szarkowska, Agnieszka, Łukasz Dutka, Anna Szychowska & Olga Pilipczuk
2018. Visual attention distribution in intralingual respeaking. In Eye tracking and multidisciplinary studies on translation [Benjamins Translation Library, 143], ► pp. 185 ff. DOI logo
Kruger, Haidee & Jan‐Louis Kruger
2017. Cognition and Reception. In The Handbook of Translation and Cognition, ► pp. 71 ff. DOI logo

This list is based on CrossRef data as of 4 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue