Cover not available

Article published In: Methodological Issues in Experimental Research in Audiovisual Translation and Media Accessibility
Edited by Gian Maria Greco, Anna Jankowska and Agnieszka Szarkowska
[Translation Spaces 11:1] 2022
► pp. 12–37

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Establishing a theoretical framework for AVT research

The importance of cognitive models

ORCiD logo with link | Macquarie University | North-West University
ORCiD logo with link | Macquarie University
Published online: 19 July 2022
DOI logo with link
https://doi.org/10.1075/ts.21024.kru
Abstract
Empirical research on the cognitive processing of audiovisual translation (AVT) products has been thriving over the past decade. While the use of cutting-edge experimental tools such as eye trackers has drawn increasing scholarly attention and accelerated the progress in understanding the complex mental processes involved in the reception of multimodal AVT products, relatively less attention has been devoted to the importance of establishing a theoretical framework or cognitive model that can explain and predict the behaviours observed in empirical experiments. By reviewing numerous theories or cognitive models relevant to AVT research in explaining how different perceptual and cognitive systems operate for understanding multimodal products, this paper calls for engagement with these theoretical frameworks and models to work towards a robust model that can generate testable hypotheses for the integration and interaction of multiple sources of information involved in the processing of AVT or other multimodal products.
Article outline
  • Introduction
  • Memory and cognition
    • Modal model – Atkinson and Shiffrin (1968)
    • Model of working memory – Baddeley and Hitch (1974); Baddeley (2000, 2010, 2012)
  • Multimodality
    • Cognitive theory of multimedia learning (CTML) – Mayer (2014)
      • Dual-channel assumption
      • Limited capacity assumption
      • Active processing assumption
    • Integrated model of text and picture comprehension (ITPC) – Schnotz and Bannert (2003)
    • Scene perception and event comprehension theory (SPECT) – Loschky et al. (2020)
      • Front-end cognitive processes
      • Back-end cognitive processes
      • Dynamic visual narratives
  • Models in reading research
    • The E-Z Reader model – Reichle et al. (2012)
    • The Über-Reader model – Reichle (2021)
    • Multimodal integrated-language framework (MILF) – Liao et al. (2021a)
  • Concluding remarks
  • Acknowledgement
  • Notes
  • References
References (51)
References
Allen, Richard J., Graham J. Hitch, Judit Mate, and Alan D. Baddeley. 2012. “Feature Binding and Attention in Working Memory: A Resolution of Previous Contradictory Findings.” Quarterly Journal of Experimental Psychology 65 (12): 2369–83. Google Scholar logo with link to Google Scholar
Ans, Bernard, Serge Carbonnel, and Sylviane Valdois. 1998. “A Connectionist Multiple-Trace Memory Model for Polysyllabic Word Reading.” Psychological Review 105 (4): 678–723. Google Scholar logo with link to Google Scholar
Atkinson, Richard. C., and Richard M. Shiffrin. 1968. “Human Memory: A Proposed System and Its Control Processes.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Spence, Kenneth W., and Janet Taylor Spence, 89–195. New York: Academic Press.Google Scholar logo with link to Google Scholar
Baddeley, Alan David. 2000. “The Episodic Buffer: A New Component of Working Memory?” Trends in Cognitive Sciences 4 (11): 417–23. Google Scholar logo with link to Google Scholar
. 2010. “Working memory.” Current biology 20 (4), 136–140. Google Scholar logo with link to Google Scholar
. 2012. “Working Memory: Theories, Models, and Controversies.” Annual Review of Psychology 63 (1): 1–29. Google Scholar logo with link to Google Scholar
Baddeley, Alan David, and Graham J. Hitch. 1974. “Working memory.” In The Psychology of Learning and Motivation: Advances in Research and Theory, edited by Bower, Gordon H., 47–89. New York: Academic Press.Google Scholar logo with link to Google Scholar
Baddeley, Alan David, Richard J. Allen, and Graham J. Hitch. 2011. “Binding in Visual Working Memory: The Role of the Episodic Buffer.” Neuropsychologia 49 (6): 1393–1400. Google Scholar logo with link to Google Scholar
Chung, Jing-mei. 1999. “The Effects of Using Video Texts Supported with Advance Organizers and Captions on Chinese College Students’ Listening Comprehension: An Empirical Study.” Foreign Language Annals 32 (3): 295–308. Google Scholar logo with link to Google Scholar
Danan, Martine. 2004. “Captioning and Subtitling: Undervalued Language Learning Strategies.” Meta 49 (1): 67–77. Google Scholar logo with link to Google Scholar
de Pedro Ricoy, Raquel. 2012. “Multimodality in Translation: Steps towards Socially Useful Research.” Journal Multimodal Communication 1 (2): 181–203. Google Scholar logo with link to Google Scholar
Díaz Cintas, Jorge, and Agnieszka Szarkowska. 2020. “Introduction: Experimental Research in Audiovisual Translation–Cognition, Reception, Production.” The Journal of Specialised Translation, 331, 3–16.Google Scholar logo with link to Google Scholar
Forstmann, Birte U., Eric-Jan Wagenmakers, Tom Eichele, Scott Brown, and John T. Serences. 2011. “Reciprocal Relations between Cognitive Neuroscience and Formal Cognitive Models: Opposites Attract?” Trends in Cognitive Sciences 15 (6): 272–79. Google Scholar logo with link to Google Scholar
Gernsbacher, Morton Ann. 2015. “Video Captions Benefit Everyone.” Edited by Susan T. Fiske. Policy Insights from the Behavioral and Brain Sciences 2 (1): 195–202. Google Scholar logo with link to Google Scholar
Hintzman, Douglas L. 1984. “MINERVA 2: A Simulation Model of Human Memory.” Behavior Research Methods, Instruments, & Computers 16 (2): 96–101. Google Scholar logo with link to Google Scholar
Kaindl, Klaus. 2013. “Multimodality and Translation.” In The Routledge Handbook of Translation Studies, edited by Millán, Carmen, and Francesca Bartrina, 257–269. London: Routledge.Google Scholar logo with link to Google Scholar
Kalyuga, Slava, and John Sweller. 2014. “The Redundancy Principle in Multimedia Learning.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 247–262. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
Ketola, Anne. 2016. “Towards a Multimodally Oriented Theory of Translation: A Cognitive Framework for the Translation of Illustrated Technical Texts.” Translation Studies 9 (1): 67–81. Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis. 2016. “Psycholinguistics and Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 276–87. Google Scholar logo with link to Google Scholar
. 2019. “Eye tracking in Audiovisual Translation Research.” In The Routledge Handbook of Audiovisual Translation, edited by Pérez-González, Luis, 350–366. London: Routledge, Taylor and Francis Group.Google Scholar logo with link to Google Scholar
. 2020. “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Alves, Fabio, and Arnt Lykke Jakobsen, 433–444. London, UK: Routledge. Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis, and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–20. Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis, Natalia Wisniewska, and Sixin Liao. 2021. “Why Subtitle Speed Matters: Evidence from Word Skipping and Rereading.” Applied Psycholinguistics 43 (1): 211–36. Google Scholar logo with link to Google Scholar
Liao, Sixin, and Jan-Louis Kruger. 2022. “Cognitive Processing of Subtitles: Charting the Future by Mapping the Past.” In The Routledge Handbook of Translation, Interpreting, and Bilingualism, edited by Ferreira, Aline, and John W. Schwieter.Google Scholar logo with link to Google Scholar
Liao, Sixin, Lili Yu, Jan-Louis Kruger, and Erik D. Reichle. 2021a. “The impact of Audio on the Reading of Inter- vs. Intra-lingual Subtitles: Evidence from Eye Movements.” Applied Psycholinguistics 43 (1): 237–69. Google Scholar logo with link to Google Scholar
Liao, Sixin, Lili Yu, Erik D. Reichle, and Jan-Louis Kruger. 2021b. “Using Eye Movements to Study the Reading of Subtitles in Video.” Scientific Studies of Reading 25(5): 417–435. Google Scholar logo with link to Google Scholar
Loschky, Lester C., Adam M. Larson, Tim J. Smith, and Joseph P. Magliano. 2020. “The Scene Perception & Event Comprehension Theory (SPECT) Applied to Visual Narratives.” Topics in Cognitive Science 12 (1): 311–51. Google Scholar logo with link to Google Scholar
Lum, Jarrad A. G., Gina Conti-Ramsden, Debra Page, and Michael T. Ullman. 2012. “Working, Declarative and Procedural Memory in Specific Language Impairment.” Cortex 48 (9): 1138–54. Google Scholar logo with link to Google Scholar
Mayer, Richard E. 2014. The Cambridge Handbook of Multimedia Learning. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
Paivio, Allan. 1986. Mental Representations: A Dual Coding Approach. Oxford, England: Oxford University Press.Google Scholar logo with link to Google Scholar
. 1991. “Dual Coding Theory: Retrospect and Current Status.” Canadian Journal of Psychology 45 (3): 255–87. Google Scholar logo with link to Google Scholar
. 2007. Mind and Its Evolution: A Dual Coding Theoretical Approach. Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Perego, Elisa, Fabio Del Missier, Marco Porta, and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13 (3): 243–72. Google Scholar logo with link to Google Scholar
Pérez-González, Luis. 2014. “Multimodality in Translation and Interpreting Studies: Theoretical and Methodological Perspectives.” In A Companion to Translation Studies, edited by Bermann, Sandra, and Catherine Porter, 119–131. Malden, MA/Oxford, England: Wiley-Blackwell. Google Scholar logo with link to Google Scholar
Prezenski, Sabine, André Brechmann, Susann Wolff, and Nele Russwinkel. 2017. “A Cognitive Modeling Approach to Strategy Formation in Dynamic Decision Making.” Frontiers in Psychology 81 (August): 1–18. Google Scholar logo with link to Google Scholar
Reichle, Erik D. 2021. Computational Models of Reading: A Handbook. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Reichle, Erik D., Alexander Pollatsek, and Keith Rayner. 2012. “Using E-Z Reader to Simulate Eye Movements in Nonreading Tasks: A Unified Framework for Understanding the Eye–Mind Link.” Psychological Review 119 (1): 155–85. Google Scholar logo with link to Google Scholar
Reichle, Erik D., Lili Yu, Sixin Liao, and Jan-Louis Kruger. 2021. “Using Simulations to Understand the Reading of Rapidly Displayed Subtitles.” Proceedings of the 43rd Annual Conference of the Cognitive Science Society.Google Scholar logo with link to Google Scholar
Schnotz, Wolfgang. 2005. “Integrated Model of Text and Picture Comprehension.” In The Cambridge Handbook of Multimedia Learning, edited by Mayer, Richard E., 49–70. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
Schnotz, Wolfgang, and Maria Bannert. 2003. “Construction and Interference in Learning from Multiple Representation.” Learning and Instruction 13 (2): 141–56. Google Scholar logo with link to Google Scholar
Schnotz, Wolfgang. 2014. “Integrated model of text and picture comprehension.” In The Cambridge Handbook of Multimedia Learning (2nd ed), edited by Mayer, Richard E., 72–103. Google Scholar logo with link to Google Scholar
Schnotz, Wolfgang, and Inga Wagner. 2018. “Construction and Elaboration of Mental Models through Strategic Conjoint Processing of Text and Pictures.” Journal of Educational Psychology 110 (6): 850–63. Google Scholar logo with link to Google Scholar
Seeber, Kilian G. 2017. Multimodal processing in simultaneous interpreting. In The Handbook of Translation and Cognition, edited by Schwieter, John and Aline Ferreira, 461–475. Joh Wiley & Sons. Google Scholar logo with link to Google Scholar
Squire, Larry R., and Stuart M. Zola. 1996. “Structure and Function of Declarative and Nondeclarative Memory Systems.” Proceedings of the National Academy of Sciences 93 (24): 13515–22. Google Scholar logo with link to Google Scholar
Szarkowska, Agnieszka, and Piotr Wasylczyk. 2018. “Five Things You Wanted to Know about Audiovisual Translation Research, but Were Afraid to Ask.” Journal of Audiovisual Translation 1 (1): 8–25. Google Scholar logo with link to Google Scholar
Szarkowska, Agnieszka, Breno Silva, and David Orrego-Carmona. 2021. “Effects of Subtitle Speed on Proportional Reading Time.” Translation, Cognition & Behavior. Google Scholar logo with link to Google Scholar
Taylor, Christopher. 2016. “The Multimodal Approach in Audiovisual Translation.” Target. International Journal of Translation Studies 28 (2): 222–36. Google Scholar logo with link to Google Scholar
Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola. 2018. “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia, New Series: Themes in Translation Studies, 171, 1–21.Google Scholar logo with link to Google Scholar
Ullman, Michael T. 2001. “A Neurocognitive Perspective on Language: The Declarative/Procedural Model.” Nature Reviews Neuroscience 2 (10): 717–26. Google Scholar logo with link to Google Scholar
2004. “Contributions of Memory Circuits to Language: The Declarative/Procedural Model.” Cognition 92 (1–2): 231–70. Google Scholar logo with link to Google Scholar
Zhong, Ling, Brianne P. Noud, Harriet Pruitt, Steven C. Marcrum, and Erin M. Picou. 2021. “Effects of Text Supplementation on Speech Intelligibility for Listeners with Normal and Impaired Hearing: A Systematic Review with Implications for Telecommunication.” International Journal of Audiology, June, 1–11. Google Scholar logo with link to Google Scholar
Cited by (3)

Cited by three other publications

Hu, Bei
2025. Charting Translation Reception, DOI logo
Bolaños García-Escribano, Alejandro
2024. Cloud subtitling in research-led education. Babel. Revue internationale de la traduction / International Journal of Translation / Revista Internacional de Traducción 70:4 ► pp. 484 ff. DOI logo
Jiang, Qihang & Stephen Doherty
2024. An eye-tracking study on the cognitive processing, reception, and perception of swearword translation in films. Translation Spaces 13:2 ► pp. 293 ff. DOI logo

This list is based on CrossRef data as of 6 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue