Cover not available

Article published In:
[Translation, Cognition & Behavior 4:2] 2021
► pp. 253–280

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required
thematic section

The ‘hookability’ of multimodal impact captions

A mixed-methods exploratory study of Japanese TV viewers

ORCiD logo with link | Dublin City University
ORCiD logo with link | University of New South Wales
ORCiD logo with link | University of Auckland
Published online: 3 December 2021
DOI logo with link
https://doi.org/10.1075/tcb.00055.sas
Abstract
The use of captions has grown in recent years in both traditional and new media, particularly in terms of the diversity of style, content, and function. Impact captions have emerged as a popular form of captions for hearing viewers and contain rich multimodal information which is employed to capture viewer attention and enhance engagement, particularly in situations where there is competition for viewer attention. Drawing upon relevance theory, we argue how impact captions could effectively attract and hold visual attention owing to their balance between processing effort and contextual effects. This exploratory study employs a dual-task paradigm and uses authentic materials and viewing situations to further examine the ability of multimodal impact captions to attract and retain overt visual attention amongst a small sample of TV viewers. Our results provide novel insight into the apparent highly individualised efficacy of impact captions, where we identify several variables of interest in participants’ viewing behaviours. We conclude with a discussion of the study’s contributions, limitations, and an outline for future work.
Article outline
  • 1.Introduction
    • 1.1Impact captions
    • 1.2Theoretical framework
  • 2.Methodology
    • 2.1Quasi-experimental design
    • 2.2Procedure
  • 3.Results
    • 3.1Viewing behaviours
    • 3.2Impact captions
    • 3.3Visual attention
    • 3.4Correlations between impact captions and visual attention
    • 3.5Summary of results
    • 3.6Retention of programme content
  • 4.Discussion
  • 5.Conclusions
  • Notes
  • References
References (46)
References
Baumgarten, Nicole. 2008. “Yeah, That’s It! Verbal Reference to Visual Information in Film Texts and Film Translations.” Meta: Translators’ Journal 53 (1): 6–25. Google Scholar logo with link to Google Scholar
Bogucki, Łukasz. 2004. “The Constraint of Relevance in Subtitling.” The Journal of Specialised Translation 11: 71–88.Google Scholar logo with link to Google Scholar
Caffrey, Colm. 2008. “Using Pupilometric, Fixation-Based and Subjective Meastures to Measure of the Processing Effect Experienced when Viewing Subtitled TV Anime with Pop-Up Gloss”. In Looking at Eyes: Eye-tracking Studies of Reading and Translation Processing. Edited by Susanne Göpferich, Arnt Lykke Jakobsen, and Inger Mees, 125–144. Copenhagen: Samfundslitteratur.Google Scholar logo with link to Google Scholar
. 2009. “Relevant Abuse? Investigating the Effects of an Abusive Subtitling Procedure on the Perception of TV Anime using Eye Tracker and Questionnaire.” PhD thesis. Dublin City University.
Chung, Marvin and Nicholas Turk-Browne. 2007. “Interactions between Attention and Memory.” Current Opinion in Neurobiology 17 (2): 177–184. Google Scholar logo with link to Google Scholar
Díaz Cintas, Jorge & Remael, Aline. 2021. Subtitling: Concepts and Practices. London, UK: Routledge. Google Scholar logo with link to Google Scholar
Doherty, Stephen. 2016. “The Impact of Translation Technologies on the Process and Product of Translation”. International Journal of Communication (10): 947–969.Google Scholar logo with link to Google Scholar
. 2020. “Multisensory Integration in Audiovisual Translation”. In Multilingual Mediated Communication and Cognition. Edited by Ricardo Muñoz Martín and Sandra Halverson, 155–170. London: Routledge. Google Scholar logo with link to Google Scholar
Ekman, Papul, Wallace V. Friesen, and Joseph C. Hager. 2002. The Facial Action Coding System CD-ROM. Salt Lake City: Research Nexus.Google Scholar logo with link to Google Scholar
Fritz, Jonathan, Mounya Elhilali, Stephen David, and Shihab Shamma. 2007. “Auditory Attention: Focusing the Searchlight on Sound.” Current Opinion in Neurobiology 17 (4): 437–455. Google Scholar logo with link to Google Scholar
Gambier, Yves. 2018. “Translation Studies, Audiovisual Translation and Reception.” In Reception Studies and Audiovisual Trannslation. Edited by Elena Di Giovanni and Yves Gambier, 43–68. Ansterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Gibbs, Raymond & Markus Tendahl. 2006. “Cognitive Effort and Effects in Metaphor Comprehension: Relevance Theory and Psycholinguistics”. Mind & Language 21 (3): 379–403. Google Scholar logo with link to Google Scholar
Karamitroglou, Fotios. 1998. A Proposed Set of Subtitling Standards in Europe. [Online] Available at: [URL]
. 2000. Towards a Methodology for the Investigation of Norms in Audiovisual Translation. Amsterdam: Rodopi. Google Scholar logo with link to Google Scholar
Kennedy, Daniel, Brian D’Onofrio, Patrick Quinn, Sven Bölte, Paul Lichtenstein & Terje Falck-Ytter. 2017. “Genetic Influence on Eye Movements to Complex Scenes at Short Timescales”. Current Biology 27 (22): 3554–3560. Google Scholar logo with link to Google Scholar
Knoblauch, Kenneth, Aries Arditi, and Janet Szlyk. 1991. “Effects of Chromatic and Luminance Contrast on Reading”. Journal of the Optical Society of America 8 (2): 428–439. Google Scholar logo with link to Google Scholar
Koga, Tadamichi. 2013. “Research Report: An Attempt of Quantitative Analysis on Directing Elements in Information and Variety Programs: From an Analytical Survey on Primetime Television Programs.” The NHK Monthly Report on Broadcast Research 63 (5): 68–81.Google Scholar logo with link to Google Scholar
. 1995. “Reception of Subtitles. The Non-existent Ideal Viewer”, Translatio (Nouvelles de la FIT/FIT Newsletter) 14 (3–4), 376–383. Audiovisual Communication and Language Transfer, International Forum, Strasbourg. 22–24 June 1995 Forum, Strasbourg 22–24.Google Scholar logo with link to Google Scholar
Kress, Gunter, and Theo Van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Edward Arnold.Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis & Stephen Doherty. 2016. “Measuring Cognitive Load in the Presence of Educational Video: Towards a Multimodal Methodology”. The Australasian Journal of Educational Technology (32): 19–31. Google Scholar logo with link to Google Scholar
Lang, Annie. 1990. “Involuntary Attention and Physiological Arousal Evoked by Structural Features and Emotional Content in TV Commercials”. Communication Research 17 (3): 275–299. Google Scholar logo with link to Google Scholar
Legge, Gordon. 2006. Psychophysics of Reading. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Mangirón Hevia, Carmen. 2013. “Subtitling in Game Localisation: A Descriptive Study.” Perspectives: Studies in Translatology, 21(1): 42–56. Google Scholar logo with link to Google Scholar
Orero Clavero, Pilar, Stephen Doherty, Jan-Louis Kruger, Anna Matamala Ripoll, Jan Pedersen, Elisa Perego, Pablo Romero Fresco, Sara Rovira Esteva, Olga Soler Vilageliu & Agnieszka Szarkowska. 2018. “Conducting Experimental Research in Audiovisual Translation (AVT): A Position Paper”. The Journal of Specialised Translation (30): 105–126.Google Scholar logo with link to Google Scholar
O’Hagan, Minako. 2010. “Japanese TV Entertainment: Framing Humour with Open Caption Telop”. In Translation, Humour and the Media. Edited by Dalia Chiaro, 70–88. London: Bloomsbury.Google Scholar logo with link to Google Scholar
O’Hagan, Minako & Ryoko Sasamoto. 2016. “Crazy Japanese Subtitles? Shedding Light on the Impact of Impact Captions with a Focus on Research Methodology”. In Eyetracking and Applied Linguistics. Edited by Silvia Hansen-Schirra and Sambor Grucza, 31–58. Berlin: Language Science Press.Google Scholar logo with link to Google Scholar
Paas, Fred, Juhani Tuovinen, Huib Tabbers and Pascal Van Gerven. 2003. Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Educational Psychologist – EDUC PSYCHOL. 381. 63–71. Google Scholar logo with link to Google Scholar
Park, Joseph. 2008. “Ideological Functions of Subtitles in Multimodal Media Texts: Impact Captioning on Korean Television.” Paper presented at the 4th International Conference on Multimodality: From Print to Interactive Digital Media: Technology, Multimodal Representation and Knowledge, Singapore Management University, Singapore, July 30–August 1.
Pashler, Harold. 1994. “Dual-task Interference in Simple Tasks: Data and Theory.” Psychological Bulletin 116 (2): 220–244. Google Scholar logo with link to Google Scholar
Pérez González, Luis. 2007. “Intervention in New Amateur Subtitling Cultures: A Multimodal Account.” Linguistica Antverpiensia (6): 67–80.Google Scholar logo with link to Google Scholar
Quittner, Alexandra, Linda Smith, Mary Osberger, Teresa Mitchell, and Donald Katz. 1994. “The Impact of Audition on the Development of Visual Attention.” Psychological Science 5 (6): 347–353. Google Scholar logo with link to Google Scholar
Sasamoto, Ryoko. 2014. “Impact Caption as a Highlighting Device: Attempts at Viewer Manipulation on TV”. Discourse, Context, and Media 61: 1–10. Google Scholar logo with link to Google Scholar
Sasamoto, Ryoko & Stephen Doherty. 2016. “Towards the Optimal Use of Impact Captions on TV Programmes”. In Conflict and Communication: A Changing Asia in a Globalising World. Edited by Minako O’Hagan and Qi Zhang, 210–247. Hauppauge, NY: Nova Science Publishers.Google Scholar logo with link to Google Scholar
Sasamoto, Ryoko & Minako O’Hagan. 2020. “Relevance, Style and Multimodality: Typographical Features as Stylistic Devices”. In Relevance Theory, Figuration, and Continuity in Pragmatics. Edited by A. Piskorska, 194–228. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Sasamoto, Ryoko, Minako O’Hagan & Stephen Doherty. 2017. “Telop, Affect, and Media Design: A Multimodal Analysis of Japanese TV Programs”. Television & New Media, 18 (5): 427–440. Google Scholar logo with link to Google Scholar
Schultheis, Hoger. and Jameson, Anthony. 2004. “Assessing Cognitive Load in Adaptive Hypermedia Systems: Physiological and Behavioural Methods.” In Adaptive Hypermedia and Adaptive Web-based Systems. Edited by Paul de Bra and Wolfgang Nejdl, 18–24. Eindhoven, Netherlands: Springer Verlag. Google Scholar logo with link to Google Scholar
Shiota, Eiko. 2003. “関連性理論とテロップ の理解 [Relevance Theory and Understanding of Telop].” The Graduate School Review of the English Language and Literature 311: 63–91.Google Scholar logo with link to Google Scholar
Shitara, Kaoru. 2011. “The Change of Telops on NHK Variety Shows.” Bulletin of Mukogawa Women’s University, Humanities and Social Science 591: 1–9.Google Scholar logo with link to Google Scholar
Sperber, Dan and Deirdre Wilson. 1995. Relevance: Communication and Cognition. Oxford: Blackwells.Google Scholar logo with link to Google Scholar
. 1986/1995. Relevance: Communication and Cognition, 2nd ed. Oxford: Blackwells.Google Scholar logo with link to Google Scholar
Taylor, Christopher. 2003. “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. Google Scholar logo with link to Google Scholar
. 2004. “Multimodal Text Analysis and Subtitling”. In Perspectives on Multimodality. Edited by Eija Ventola, Cassily Charles and Martin Kaltenbacher, 153–172. Amsterdam, Netherlands: John Benjamins. Google Scholar logo with link to Google Scholar
Telebi Kiroku. 2013. Honmadekka TV Shichoritsu 2013. Telebi kiroku Available at: [URL]
Wickens, Christopher. 1991. “Processing Resources and Attention.” In Multiple-task Performance. Edited by D. Damos, 3–34. London: Taylor & Francis.Google Scholar logo with link to Google Scholar
Wilson, Deirdre & Dan Sperber. 2002. “Truthfulness and Relevance.” Mind 111 (443): 583–632. Google Scholar logo with link to Google Scholar
Cited by (3)

Cited by three other publications

Hu, Siying, Huanchen Wang, Yu Zhang, Piaohong Wang & Zhicong Lu
2025. DanModCap: Designing a Danmaku Moderation Tool for Video-Sharing Platforms that Leverages Impact Captions with Large Language Models. Proceedings of the ACM on Human-Computer Interaction 9:2 ► pp. 1 ff. DOI logo
Lahiani, Raja
2025. Smurfing beyond translation. Translation Spaces DOI logo
Zhang, Yu, Yi Wen, Siying HU & Zhicong Lu
2025. SpeechCap: Leveraging Playful Impact Captions to Facilitate Interpersonal Communication in Social Virtual Reality. Proceedings of the ACM on Human-Computer Interaction 9:7 ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 5 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue