Cover not available

Article published In: Translation, Cognition & Behavior
Vol. 4:1 (2021) ► pp.1–25

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Does it help to see the speaker’s lip movements?

An investigation of cognitive load and mental effort in simultaneous interpreting

ORCiD logo with link | Zurich University for Applied Sciences (ZHAW)
Published online: 19 May 2021
DOI logo with link
https://doi.org/10.1075/tcb.00049.gie
Abstract
Simultaneous interpreting combines auditory and visual information. Within a multitude of visual inputs that interpreters receive, the one from the speaker seems to be particularly important (Bühler, Hildegund. 1985. “Conference Interpreting: A Multichannel Communication Phenomenon.” Meta: Journal des traducteurs, 30(1): 49–54. Google Scholar logo with link to Google Scholar; Seubert, Sabine. 2019. Visuelle Informationen beim Simultandolmetschen: Eine Eyetracking-Studie. Berlin: Frank & Timme GmbH.Google Scholar logo with link to Google Scholar). One reason might be that lip movements enhance speech perception and might thus reduce cognitive load in simultaneous interpreting and hence, induce lower mental effort. This effect may be even more pronounced when noise is added to the source speech. This study was conducted to investigate cognitive load and mental effort during simultaneous interpreting (a) with and without the ability to see speaker’s lip movements, and (b) with and without interfering noise. A group of listeners was included to control for task-related effects. Mental effort and cognitive load were measured using pupillometry, interpreting accuracy measures, and subjective reports. The facilitation hypothesis for lip movements was not confirmed. However, the pupillometric data suggests that lip movements may increase arousal.
Article outline
  • 1.Introduction
    • 1.1Lip movements as visual input in simultaneous interpreting
    • 1.2Tapping into cognitive load and mental effort
  • 2.Materials and methods
    • 2.1Participants
    • 2.2Experimental stimuli
    • 2.3Apparatus
    • 2.4Procedure
  • 3.Analysis and results
    • 3.1Subjective reports
    • 3.2Interpreting accuracy
    • 3.3Pupillometry
  • 4.Discussion and conclusions
    • 4.1What is the effect of lip movements on interpreting and what do pupil sizes tell us?
    • 4.2Limitations
    • 4.3Conclusion
  • Acknowledgments
  • Notes
  • References
References (75)
References
Anderson, Linda. 1994. “Simultaneous Interpretation: Contextual and Translation Aspects.” In Benjamins Translation Library: Vol. 31. Bridging the Gap: Empirical Research in Simultaneous Interpretation. Edited by S. Lambert and B. Moser-Mercer, 101–248. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Bates, Douglas, Martin Mächler, Ben Bolker & Steve Walker. 2015. “Fitting Linear Mixed-Effects Models Using lme4.” Journal of Statistical Software 67 (1). Google Scholar logo with link to Google Scholar
Beatty, Jack. 1982. “Phasic Not Tonic Pupillary Responses Vary With Auditory Vigilance Performance.” Psychophysiology, 19(2): 167–172. Google Scholar logo with link to Google Scholar
Bernstein, Lynne E., Edward T. Auer, Sumik Takayanagi. 2004. “Auditory Speech Detection in Noise Enhanced by Lipreading.” Speech Communication, 44(1–4): 5–18. Google Scholar logo with link to Google Scholar
Brancazio, Lawrence, Catherine T. Best and Carol A. Fowler. 2006. “Visual Influences on Perception of Speech and Nonspeech Vocal-Tract Events.” Language and Speech, 49(1): 21–53. Google Scholar logo with link to Google Scholar
Brown, Robert and Howard. E. Page. 1939. “Pupil Dilatation and Dark Adaptation.” Journal of Experimental Psychology, 25(4): 347–360. Google Scholar logo with link to Google Scholar
Bühler, Hildegund. 1985. “Conference Interpreting: A Multichannel Communication Phenomenon.” Meta: Journal des traducteurs, 30(1): 49–54. Google Scholar logo with link to Google Scholar
Chapman, C. Richard, Shunichi Oka, David H. Bradshaw, Robert C. Jacobson and Gary W. Donaldson. 1999. “Phasic Pupil Dilation Response to Noxious Stimulation in Normal Volunteers: Relationship to Brain Evoked Potentials and Pain Report.” Psychophysiology, 36(1): 44–52. Google Scholar logo with link to Google Scholar
Christensen, Rune H. B. 2015. Ordinal – Regression Models for Ordinal Data. Version 2015.1–21. [URL] (accessed 01 December 2015).
Courtney, Christopher G., Michael E. Dawson, Anne M. Schell, Arvind Iyer, Thomas D. Parsons. 2010. “Better than the Real Thing: Eliciting fear with Moving and Static Computer-generated Stimuli.” International Journal of Psychophysiology, 78(2): 107–114. Google Scholar logo with link to Google Scholar
Davies, M. 2008. Word frequency data. Retrieved from The Corpus of Contemporary American English (COCA): [URL] (accessed 19 March 2021).
Díaz-Galaz, Stephanie, Presentacion Padilla Benítez, and M. Teresa Bajo Molina. 2015. “The Role of Advance Preparation in Simultaneous Interpreting: A Comparison of Professional Interpreters and Interpreting Students”. Interpreting 17(1): 1–25. Google Scholar logo with link to Google Scholar
Dillinger, Mike. 1990. “Comprehension during Interpreting: What Do Interpreters Know that Bilinguals Don’t?” The Interpreters’ Newsletter, 31: 41–58. Accessed March 03 17, 2021. [URL]
Ehrensberger-Dow, Maureen, Michaela Albl-Mikasa, Katrin Andermatt, Andrea Hunziker Heeb and Caroline Lehr. 2020. “Cognitive Load in Processing ELF: Translators, Interpreters, and Other Multilinguals.” Journal of English as a Lingua Franca 9(2): 217–238. Google Scholar logo with link to Google Scholar
Engelhardt, Paul E., Fernanda Ferreira and Elana G. Patsenko. 2010. “Pupillometry Reveals Processing Load during Spoken Language Comprehension.” Quarterly Journal of Experimental Psychology, 63(4): 639–645. Google Scholar logo with link to Google Scholar
European Commission. 2009a. “United Airlines rewards fittest people.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission. 2009b. “Disenchantment at work.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021)
. 2012a. “Demographic shift in Europe.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
European Commission. 2012b. “Greece in the doldrums.” Retrieved from Speech Repository: [URL] (accessed 4 February 2021).
Fisk, Angus S., Shu K. E. Tam, L. A. Brown, Vladyslav V. Vyazovskiy, David M. Bannerman and Stuart N. Peirson. 2018. “Light and Cognition: Roles for Circadian Rhythms, Sleep, and Arousal.” Frontiers in Neurology, 91: 56. Google Scholar logo with link to Google Scholar
Gerver, David. 2002. “The Effects of Source Language Presentation Rate on the Performance of Simultaneous Conference Interpreters.” The Interpreting Studies. Reader. Edited by F. Pöchhacker and M. Shesinger, 52–66. London: Routledge.Google Scholar logo with link to Google Scholar
. 1974. “The Effects of Noise on the Performance of Simultaneous Interpreters: Accuracy of Performance.” Acta Psychologica, 38(3): 159–167. Google Scholar logo with link to Google Scholar
. 2018. “The Impact of Visual Input on Cognitive Load in Simultaneous Interpreting”. Mainz: Johannes Gutenberg Universität.Google Scholar logo with link to Google Scholar
Granholm, Eric, Robert F. Asarnow, Andrew J. Sarkin and Karen, L. 1996. “Pupillary Responses Index Cognitive Resource Limitations.” Psychophysiology, 331: 457–461. Google Scholar logo with link to Google Scholar
Halverson, Sandra. 2017. “Multimethod approaches.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 195–212. Hoboken: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Hild, Adelina. 2015. “Discourse Comprehension in Simultaneous Interpreting: The Role of Expertise and Redundancy.” Psycholinguistic and Cognitive Inquiries into Translation and Interpreting. Edited by A. Ferreira and J. W. Schwieter. 67–100. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Hyönä, Jukka, Jorma Tommola, Anna-Maria Alaja. 1995. “Pupil Dilation as a Measure of Processing Load in Simultaneous Interpretation and Other Language Tasks.” The Quarterly Journal of Experimental Psychology Section A, 48(3): 598–612. Google Scholar logo with link to Google Scholar
Ivars, Amparo J., Daniel P. Calatayud. 2001. “‘I Failed because I Got Nervous’. Anxiety and Performance in Interpreter Trainees: An Empirical Study.” The Interpreter’s Newsletter, 111: 105–120. Accessed March 17, 2021. [URL]
Jesse, Alexandra, Nick Vrignaud, Michael M. Cohen and Dominic W. Massaro. 2000. “The Processing of Information from Multiple Sources in Simultaneous Interpreting.” Interpreting, 5(2): 95–115. Google Scholar logo with link to Google Scholar
Jordan, Timothy R. and Paul Sergeant. 2000. “Effects of Distance on Visual and Audiovisual Speech Recognition.” Language and Speech, 43(1): 107–124. Google Scholar logo with link to Google Scholar
Kahneman, Daniel. 1973. Attention and Effort. Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar logo with link to Google Scholar
Korpal, Paweł. 2016. “Interpreting as a Stressful Activity: Physiological Measures of Stress in Simultaneous Interpreting.” Poznan Studies in Contemporary Linguistics, 52(2): 297–316. Google Scholar logo with link to Google Scholar
Kramer, Sophia E., Theo S. Kapteyn, Josst M. Festen, and Dirk J. Kuik. 1997. “Assessing Aspects of Auditory Handicap by Means of Pupil Dilation.” Audiology, 361: 155–164. Google Scholar logo with link to Google Scholar
Krejtz, Krzysztof, Andrew T. Duchowski, Anna Niedzielska, Cezary Biele, and Izabela Krejtz. 2018. “Eye Tracking Cognitive Load Using Pupil Diameter and Microsaccades with Fixed Gaze”. PLOS ONE 13(9): e0203629. Google Scholar logo with link to Google Scholar
Kuchinsky, Stefanie E., Jayne B. Ahlstrom, Kenneth I. Vaden, Stephanie L. Cute, Larry E. Humes, Judy R. Dubno and Mark A. Eckert. 2013. “Pupil Size Varies with Word Listening and Response Selection Difficulty in Older Adults with Hearing Loss: Pupil Size in Older Adults.” Psychophysiology, 50(1): 23–34. Google Scholar logo with link to Google Scholar
Lecumberri, Maria L. G., Martin Cooke and Anne Cutler. 2010. “Non-native Speech Perception in Adverse Conditions: A Review.” Speech Communication, 52(1): 864–886. Google Scholar logo with link to Google Scholar
Lee, Jieun. 2008. “Rating Scales for Interpreting Performance Assessment.” The Interpreter and Translator Trainer, 2(2): 165–184. Google Scholar logo with link to Google Scholar
Liu, Minhua, Diane L. Schallert, and Patrick J. Carroll. 2004. ‘Working Memory and Expertise in Simultaneous Interpreting’. Interpreting 6(1): 19–42. Google Scholar logo with link to Google Scholar
Lowenstein, Otto, Richard Feinberg and Irene Loewenfeld. 1963. “Pupillary Movements During Acute and Chronic Fatigue A New Test for the Objective Evaluation of Tiredness.” Investigative Ophthalmology, 2(2): 138–158.Google Scholar logo with link to Google Scholar
Massaro, Dominic W. And Michael M. Cohen. 1999. “Speech Perception in Perceivers with Hearing Loss: Synergy of Multiple Modalities.” Journal of Speech, Language, and Hearing Research, 42(1), 21–41. Google Scholar logo with link to Google Scholar
Mirman, Daniel. 2014. Growth Curve Analysis and Visualization Using R. Boca Raton, FL: CRC Press.Google Scholar logo with link to Google Scholar
Moser-Mercer, Barbara, Alexandra Künzli and Marina Korac. 1998. “Prolonged Turns in Interpreting: Effects on Quality, Physiological and Psychological Stress (Pilot study).” Interpreting, 3(1): 47–64. Google Scholar logo with link to Google Scholar
Oliva, Manuel and Andrey Anikin. 2018. “Pupil Dilation Reflects the Time Course of Emotion Recognition in Human Vocalizations.” Scientific Reports, 81, 4871. Google Scholar logo with link to Google Scholar
Peelle, Jonathan E. and Mitchell S. Sommers. 2015. “Prediction and Constraint in Audiovisual Speech Perception.” Cortex, 681: 169–181. Google Scholar logo with link to Google Scholar
Peirce, Jonathan, Jeremy R. Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Eric Kastman and Jonas Kristoffer Lindeløv. 2019. “PsychoPy2: Experiments in Behavior made Easy.” Behavior Research Methods, 51(1): 195–203. Google Scholar logo with link to Google Scholar
Peirce, Jonathan W. 2007. “PsychoPy – Psychophysics Software in Python.” Journal of Neuroscience Methods, 162(1–2), 8–13. Google Scholar logo with link to Google Scholar
R Core Team. 2016. “R: A language and environment for statistical computing.” Version 3.2.4. Retrieved from R Foundation for Statistical Computing: [URL] (accessed 1 March 2016).
Rennert, Sylvie. 2008. “Visual Input in Simultaneous Interpreting.” Meta: journal des traducteurs, 53(1): 204–217. Google Scholar logo with link to Google Scholar
Seeber, Kilian G. 2017. “Multimodal Processing in Simultaneous Interpreting.” The Handbook of Translation and Cognition. Edited by J. Schwieter and A. Ferreira. 461–475. Hoboken: Wiley Blackwell. Google Scholar logo with link to Google Scholar
2015. “Cognitive Load in Simultaneous Interpreting: Measures and Methods.” Interdisciplinarity in translation and interpreting process research. Edited by M. Ehrensberger-Dow, S. Göpferich & S. O’Brien (Eds.). 18–33. Google Scholar logo with link to Google Scholar
Seeber, Kilian G. and Dirk Kerzel. 2012. “Cognitive Load in Simultaneous Interpreting: Model Meets Data.” International Journal of Bilingualism, 16(2): 228–242. Google Scholar logo with link to Google Scholar
Seeber, Kilian G. 2011. “Cognitive Load in Simultaneous Interpreting.” Interpreting, 13(2): 176–204. Google Scholar logo with link to Google Scholar
Seubert, Sabine. 2019. Visuelle Informationen beim Simultandolmetschen: Eine Eyetracking-Studie. Berlin: Frank & Timme GmbH.Google Scholar logo with link to Google Scholar
Stachowiak-Szymczak, Katarzyna. 2019. Eye Movements and Gestures in Simultaneous and Consecutive Interpreting. Google Scholar logo with link to Google Scholar
Sumby, William H. and Irwin Pollack. 1954. “Visual Contribution to Speech Intelligibility in Noise.” Journal of the Acoustical Society of America, (261): 212–215. Google Scholar logo with link to Google Scholar
Tabri, Dollen, Kim M. S. A. Chacra & Tim Pring. 2010. “Speech Perception in Noise by Monolingual, Bilingual and Trilingual Listeners.” International Journal of Language & Communication Disorders, 46(4): 411–422. Google Scholar logo with link to Google Scholar
Thomas, Sharon M., Timothy R. Jordan. 2004. “Contributions of Oral and Extraoral Facial Movement to Visual and Audiovisual Speech Perception.” Journal of Experimental Psychology: Human Perception and Performance, 30(5): 873–888. Google Scholar logo with link to Google Scholar
Tobii support. 2016, July 20. Personal communication.Google Scholar logo with link to Google Scholar
Tobii Technology. 2010. Tobii TX300 Eye Tracker. User Manual. Accessed March 17, 2021. [URL]
Tommola, Jorma and Johan Lindholm. 1995. “Experimental Research on Interpreting: Which Dependent Variable?” Topics in interpreting research. Edited by J. Tommola. 121–133. Turku: University of Turku.Google Scholar logo with link to Google Scholar
Tye-Murray, Nancy, Mitchell Sommers and Brent Spehar. 2007. “Auditory and Visual Lexical Neighborhoods in Audiovisual Speech Perception.” Trends in Amplification, 11(4): 233–241. Google Scholar logo with link to Google Scholar
van der Wel, Pauline and Henk van Steenbergen. 2018. “Pupil Dilation as an Index of Effort in Cognitive Control Tasks: A Review.” Psychonomic Bulletin & Review, 25(6): 2005–2015. Google Scholar logo with link to Google Scholar
Vatikiotis-Bateson, Eric, Inge-Marie Eigsti, Sumio Yano and Kevin G. Munhall. 1998. “Eye Movement of Perceivers during Audiovisual Speech Perception.” Perception & Psychophysics, 60(6): 926–940. Google Scholar logo with link to Google Scholar
von Kriegstein, Katharina, Özgür Dogan, Martina Grüter, Anne-Lise Giraud, Christian A. Kell, Thomas Grüter, Andreas Kleinschmidt and Stefan A. Kiebel. 2008. “Simulation of Talking Faces in the Human Brain Improves Auditory Speech Recognition.” Proceedings of the National Academy of Sciences, 105(18): 6747–6752. Google Scholar logo with link to Google Scholar
Wickham, H. 2009. ggplot2: Elegant Graphics for Data Analysis. New York: Springer. Google Scholar logo with link to Google Scholar
Wong, Hoe K. and Julien Epps. 2016. “Pupillary Transient Responses to Within-Task Cognitive Load Variation.” Computer Methods and Programs in Biomedicine, 1371: 47–63. Google Scholar logo with link to Google Scholar
Yoon, Jinny J. and Helen V. Danesh-Meyer. 2019. “Caffeine and the Eye.” Survey of Ophthalmology, 64(3): 334–344. Google Scholar logo with link to Google Scholar
Zekveld, Adriana A., Dirk J. Heslenfeld, Ingrid S. Johnsrude, Niek J. Versfeld and Sophia E. Kramer. 2014. “The Eye as a Window to the Listening Brain: Neural Correlates of Pupil Size as a Measure of Cognitive Listening Load.” NeuroImage, 1011: 76–86. Google Scholar logo with link to Google Scholar
Zwischenberger, Cornelia. 2010. “Quality Criteria in Simultaneous Interpreting: An International vs. A National View.” The Interpreters’ Newsletter, 151: 127–142. Accessed 17 March 2021. [URL]
Cited by (7)

Cited by seven other publications

Ahrens, Barbara & Przemysław Janikowski
2025. Speech recording. In Research Methods in Cognitive Translation and Interpreting Studies [Research Methods in Applied Linguistics, 10], ► pp. 183 ff. DOI logo
Shao, Zhangminzi & Bart Defrancq
2025. Fundamental frequency as an acoustic mirror of interpreters’ cognitive states. Interpreting. International Journal of Research and Practice in Interpreting 27:1 ► pp. 1 ff. DOI logo
Bauer, Robert & Petra Jansen
2024. A short mindfulness induction might increase women’s mental rotation performance. Consciousness and Cognition 123 ► pp. 103721 ff. DOI logo
Gieshoff, Anne Catherine & Michaela Albl-Mikasa
2024. Interpreting accuracy revisited: a refined approach to interpreting performance analysis. Perspectives 32:2 ► pp. 210 ff. DOI logo
Peng, Xuejiao, Xiangling Wang & Guangjiao Chen
2024. Text availability and the speaker’s visibility in simultaneous interpreting: effects on the process, product, and interpreters’ perceptions. Perspectives ► pp. 1 ff. DOI logo
Gieshoff, Anne Catherine & Andrea Hunziker Heeb
2023. Cognitive load and cognitive effort. Translation, Cognition & Behavior 6:1 ► pp. 3 ff. DOI logo
Melicherčíková, Miroslava & Soňa Hodáková
2023. Prieniky a odlišnosti v kognitívno-osobnostnom profile a tlmočníckom výkone profesionálov a študentov, DOI logo

This list is based on CrossRef data as of 5 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue