Cover not available

Article published In: Target
Vol. 32:1 (2020) ► pp.37–58

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Multimodal processing in simultaneous interpreting with text

Interpreters focus more on the visual than the auditory modality

ORCiD logo with link | Adam Mickiewicz University in Poznań
 | University of Silesia in Katowice
 | Adam Mickiewicz University in Poznań
Published online: 21 January 2020
DOI logo with link
https://doi.org/10.1075/target.18157.chm
Abstract
The present study focuses on (in)congruence of input between the visual and the auditory modality in simultaneous interpreting with text. We asked twenty-four professional conference interpreters to simultaneously interpret an aurally and visually presented text with controlled incongruences in three categories (numbers, names and control words), while measuring interpreting accuracy and eye movements. The results provide evidence for the dominance of the visual modality, which goes against the professional standard of following the auditory modality in the case of incongruence. Numbers enjoyed the greatest accuracy across conditions possibly due to simple cross-language semantic mappings. We found no evidence for a facilitation effect for congruent items, and identified an impeding effect of the presence of the visual text for incongruent items. These results might be interpreted either as evidence for the Colavita effect (in which visual stimuli take precedence over auditory ones) or as strategic behaviour applied by professional interpreters to avoid risk.
Article outline
  • 1.Introduction
  • 2.Existing research on multimodal processing
    • 2.1Multimodal processing in simultaneous interpreting with text
    • 2.2Cognitive aspects of multimodal processing
  • 3.The present study
    • 3.1Participants
    • 3.2Materials
    • 3.3Procedure
  • 4.Results
  • 5.Discussion
  • 6.Conclusions
  • References
References (48)
References
Braun, Susanne, and Andrea Clarici. 1996. “Inaccuracy for Numerals in Simultaneous Interpretation: Neurolinguistic and Neuropsychological Perspectives.” Interpreting 71: 85–102.Google Scholar logo with link to Google Scholar
Cammoun, Rawdha, Catherine Davies, Konstantin Ivanov, and Boris Naimushin. 2009. Simultaneous Interpretation with Text. Is the Text ‘Friend’ or ‘Foe’? Laying Foundations for a Teaching Module.” MA diss. University of Geneva.Google Scholar logo with link to Google Scholar
Chmiel, Agnieszka. 2015. “Przetwarzanie w tłumaczeniu symultanicznym.” In Dydaktyka tłumaczenia ustnego, edited by Agnieszka Chmiel and Przemysław Janikowski, 227–247. Katowice: Stowarzyszenie Inicjatyw Wydawniczych przy współpracy UŚ i UAM.Google Scholar logo with link to Google Scholar
Colavita, Francis. 1974. “Human Sensory Dominance.” Perception & Psychophysics 16 (2): 409–412. Google Scholar logo with link to Google Scholar
Dehaene, Stanislas, and Laurent Cohen. 2007. “Cultural Recycling of Cortical Maps.” Neuron 56 (2): 384–398. Google Scholar logo with link to Google Scholar
Desmet, Bart, Mieke Vandierendonck, and Bart Defrancq. 2018. “Simultaneous Interpretation of Numbers and the Impact of Technological Support.” In Interpreting and Technology, edited by Claudio Fantinuoli, 13–27. Berlin: Language Science Press.Google Scholar logo with link to Google Scholar
Deutsch, Avital, and Shlomo Bentin. 2001. “Syntactic and Semantic Factors in Processing Gender Agreement in Hebrew: Evidence from ERPs and Eye Movements.” Journal of Memory and Language 45 (2): 200–224. Google Scholar logo with link to Google Scholar
Doehrmann, Oliver, and Marcus Naumer. 2008. “Semantics and the Multisensory Brain: How Meaning Modulates Processes of Audio-Visual Integration.” Brain Research 12421 (November): 136–150. Google Scholar logo with link to Google Scholar
Gile, Daniel. 1997. “Conference Interpreting as a Cognitive Management Problem.” In Cognitive Processes in Translation and Interpreting, edited by Joseph Danks, Gregory Shreve, Stephen Fountain, and Michael McBeath, 196–214. Thousand Oaks, CA: Sage.Google Scholar logo with link to Google Scholar
. 2018. “The Effort Models and Gravitational Model Clarifications and Update.” June 1. [URL]
Hirst, Rebecca, Lucy Cragg, and Harriet Allen. 2018. “Vision Dominates Audition in Adults but Not Children: A Meta-Analysis of the Colavita Effect.” Neuroscience & Biobehavioral Reviews 941 (November): 286–301. Google Scholar logo with link to Google Scholar
Iodice, Rosario, Juan José García Meilán, Juan Carro Ramos, and Jeff A. Small. 2018. “Sentence Context and Word-Picture Cued-Recall Paired-Associate Learning Procedure Boosts Recall in Normal and Mild Alzheimer’s Disease Patients.” Behavioural Neurology. Google Scholar logo with link to Google Scholar
Ivanov, Konstantin, Kate Davies, and Boris Naimushin. 2014. “Teaching Simultaneous Interpretation with Text.” In Взаимопонимание в Многоязычном Мире: Культура, Язык, Перевод. Сборник Статей в Честь Профессора и.с. Алексеевой / Fighting the Fog in Multilingualism. A Festschrift in Honour of Irina S. Alekseeva, edited by Sergey Goncharov and Angelique Antonova, 38–48. St. Petersburg: Herzen University Press.Google Scholar logo with link to Google Scholar
Janikowski, Przemysław, Agnieszka Chmiel, and Agnieszka Lijewska. (In preparation). “Coping with Sensory Overload in Simultaneous Interpreting with Text: Examining the Timecourse with Eye-Ear, Ear-Voice, and Eye-Voice Span Measures.”
Jiménez Ivars, Maria Amparo. 1999. La traducción a la vista: Un análisis descriptivo. PhD diss. Universitat Jaume I.Google Scholar logo with link to Google Scholar
Koppen, Camille, Agnès Alsius, and Charles Spence. 2008. “Semantic Congruency and the Colavita Visual Dominance Effect.” Experimental Brain Research 184 (4): 533–546. Google Scholar logo with link to Google Scholar
Koppen, Camille, and Charles Spence. 2007. “Spatial Coincidence Modulates the Colavita Visual Dominance Effect.” Neuroscience Letters 417 (2): 107–111. Google Scholar logo with link to Google Scholar
Korpal, Paweł, and Katarzyna Stachowiak. 2015. “The Visual or the Aural: Which Modality Is Dominant in Simultaneous Interpreting?” Conference paper presented at the ICEAL (International Conference on Eyetracking and Applied Linguistics), Warszawa, September 22.
. 2017. “A Closer Look at Numbers in Simultaneous Interpreting: An Eye-Tracking Study.” Poster presented at the 19th European Conference on Eye Movements, Wuppertal, August 20.
Lamberger-Felber, Heike, and Julia Schneider. 2009. “Linguistic Interference in Simultaneous Interpreting with Text: A Case Study.” In Efforts and Models in Interpreting and Translation Research. A Tribute to Daniel Gile (Benjamins Translation Library 80), edited by Gyde Hansen, Andrew Chesterman, and Heidrun Gerzymisch-Arbogast, 215–236. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Lambert, Sylvie. 2004. “Shared Attention during Sight Translation, Sight Interpretation and Simultaneous Interpretation.” Meta: Journal des traducteurs / Meta: Translators’ Journal 49 (2): 294–306. Google Scholar logo with link to Google Scholar
Laurienti, Paul, Robert Kraft, Joseph Maldjian, Jonathan Burdette, and Mark Wallace. 2004. “Semantic Congruence Is a Critical Factor in Multisensory Behavioral Performance.” Experimental Brain Research 158 (4): 405–414. Google Scholar logo with link to Google Scholar
Leeson, Loraine. 2005. “Making the Effort in Simultaneous Interpreting: Some Considerations for Signed Language Interpreters.” In Topics in Signed Language Interpreting: Theory and Practice, edited by Terry Janzen, 51–68. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Lefcheck, Jonathan, and Robert Freckleton. 2016. “PIECEWISESEM: Piecewise Structural Equation Modelling in R for Ecology, Evolution, and Systematics.” Methods in Ecology and Evolution 7 (5): 573–579. Google Scholar logo with link to Google Scholar
Lemhöfer, Kristin, and Mirjam Broersma. 2012. “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–43. Google Scholar logo with link to Google Scholar
Matusz, Pawel, Hannah Broadbent, Jessica Ferrari, Benjamin Forrest, Rebecca Merkley, and Gaia Scerif. 2015. “Multi-Modal Distraction: Insights from Children’s Limited Attention.” Cognition 1361 (March): 156–165. Google Scholar logo with link to Google Scholar
Nakagawa, Shinichi, and Holger Schielzeth. 2013. “A General and Simple Method for Obtaining R2 from Generalized Linear Mixed-Effects Models.” Methods in Ecology and Evolution 4 (2): 133–142. Google Scholar logo with link to Google Scholar
Pinochi, Diletta. 2009. “Simultaneous Interpretation of Numbers: Comparing German and English to Italian. An Experimental Study.” The Interpreters’ Newsletter 141: 33–57.Google Scholar logo with link to Google Scholar
Prior, Anat, Michal Katz, Islam Mahajna, and Orly Rubinsten. 2015. “Number Word Structure in First and Second Language Influences Arithmetic Skills.” Frontiers in Psychology 61 (March). Google Scholar logo with link to Google Scholar
Pöchhacker, Franz. 2004. Introducing Interpreting Studies. London: Routledge. Google Scholar logo with link to Google Scholar
Pym, Anthony. 2004. “Text and Risk in Translation.” In Choice and Difference in Translation: The Specifics of Transfer, edited by Maria Sidiropoulou and Anastasia Papaconstantinou, 27–42. Athens: University of Athens.Google Scholar logo with link to Google Scholar
Schweda Nicholson, Nancy. 1989. “Documentation and Text Preparation for Simultaneous Interpretation.” In Coming of Age: Proceedings of the 30th Annual Conference of the American Translators Association, Oct. 11–15, 1989, Washington, D.C., edited by Deanna Hammond, 163–182. Medford, N.J.: Learned Information Inc.Google Scholar logo with link to Google Scholar
Seeber, Kilian G. 2016. “Modeling Multimodal Processing in Simultaneous Interpreting.” Poster presented at the 75th anniversary of FTI, Genève, September 30.
Seeber, Kilian. 2017. “Multimodal Processing in Simultaneous Interpreting.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 461–475. Hoboken, NJ: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Setton, Robin, and Andrew Dawrant. 2016. Conference Interpreting: A Complete Course. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Sile, Dina. 2003. “Simultaneous Interpreting with Text: A Suggestion for Teaching Methodology.” In The Third Riga Symposium on Pragmatic Aspects of Translation: Proceedings, edited by Andrejs Veisbergs, 153–162. Riga: University of Latvia, Aarhus School of Business.Google Scholar logo with link to Google Scholar
Sinnett, Scott, Charles Spence, and Salvador Soto-Faraco. 2007. “Visual Dominance and Attention: The Colavita Effect Revisited.” Perception & Psychophysics 69 (5): 673–686. Google Scholar logo with link to Google Scholar
Sinnett, Scott, Salvador Soto-Faraco, and Charles Spence. 2008. “The Co-occurrence of Multisensory Competition and Facilitation.” Acta Psychologica 128 (1): 153–161. Google Scholar logo with link to Google Scholar
Spence, Charles. 2009. “Explaining the Colavita Visual Dominance Effect.” In Attention (Progress in Brain Research 176), edited by Narayanan Srinivasan, 245–258. Amsterdam: Elsevier. Google Scholar logo with link to Google Scholar
Spence, Charles, Cesare Parise, and Yi-Chuan Chen. 2012. “The Colavita Visual Dominance Effect.” In The Neural Bases of Multisensory Processes, edited by Micah M. Murray and Mark T. Wallace, 529–566. Boca Raton, FL: CRC Press. [URL]
Spychała, Joanna Maria. 2015. The Impact of Source Text Availability on Simultaneous Interpreting Performance. MA diss. Uniwersytet im. Adama Mickiewicza w Poznaniu.Google Scholar logo with link to Google Scholar
Stubblefield, Alexandra, Lauryn Jacobs, Yongju Kim, and Paula Goolkasian. 2013. “Colavita Dominance Effect Revisited: The Effect of Semantic Congruity.” Attention, Perception, & Psychophysics 75 (8): 1827–1839. Google Scholar logo with link to Google Scholar
Sullivan, Brian, Harm Brouwer, Alba Rodriguez, Matthew Crocker, and Pia Knoeferle. 2016. “Eye Movements during Linguistic-Visual Conflicts.” Poster presented at The Attentive Listener in the Visual World. 3rd AttLis workshop, Potsdam, March 10.
Whiting, Caroline, Yury Shtyrov, and William Marslen-Wilson. 2015. “Real-Time Functional Architecture of Visual Word Recognition.” Journal of Cognitive Neuroscience 27 (2): 246–265. Google Scholar logo with link to Google Scholar
Wille, Claudia, and Mirjam Ebersbach. 2016. “Semantic Congruency and the (Reversed) Colavita Effect in Children and Adults.” Journal of Experimental Child Psychology 1411 (January): 23–33. Google Scholar logo with link to Google Scholar
Zeier, Hans. 1997. “Psychophysiological Stress Research.” Interpreting 2 (1–2): 231–249. Google Scholar logo with link to Google Scholar
Cited by (23)

Cited by 23 other publications

Amos, Rhona & Martin J. Pickering
2025. A new perspective on models and theories of simultaneous interpreting. Target. International Journal of Translation Studies 37:2 ► pp. 159 ff. DOI logo
Chmiel, Agnieszka, Nicoletta Spinolo, Paweł Korpal, Christian Olalla-Soler, Paulina Rozkrut, Marta Kajzer-Wietrzny & Serena Ghiselli
2025. The impact of remote interpreting settings on interpreter experience and performance. Translation and Interpreting Studies DOI logo
Guo, Zidian & Ketong Li
2025. Limitations and Best Practices for the Design of Computer-Aided Interpretation Tools. In Artificial Intelligence in HCI [Lecture Notes in Computer Science, 15819], ► pp. 319 ff. DOI logo
Janikowski, Przemysław & Agnieszka Chmiel
2025. Ear–voice span in simultaneous interpreting. Interpreting. International Journal of Research and Practice in Interpreting 27:1 ► pp. 28 ff. DOI logo
Liu, Yi, Han Xu & Dechao Li
2025. Formulaicity in simultaneous interpreting: examining lexical bundles in interpreted and non-interpreted English speeches at UN Security Council meetings. Perspectives ► pp. 1 ff. DOI logo
Yang, Shanshan, Defeng Li, Victoria Lai Cheng Lei & Laura Morett
2025. Multimodal processing in simultaneous interpreting with text: Evidence from ear-eye-voice span and performance. PLOS One 20:7 ► pp. e0326527 ff. DOI logo
Li, Tianyun & Agnieszka Chmiel
2024. Automatic subtitles increase accuracy and decrease cognitive load in simultaneous interpreting. Interpreting. International Journal of Research and Practice in Interpreting 26:2 ► pp. 253 ff. DOI logo
Peng, Xuejiao, Xiangling Wang & Guangjiao Chen
2024. Text availability and the speaker’s visibility in simultaneous interpreting: effects on the process, product, and interpreters’ perceptions. Perspectives ► pp. 1 ff. DOI logo
Santamaría Urbieta, Alexandra, Elena Alcalde Peñalver & Peter Bannister
2024. Estrategias metacomunicativas y de multimodalidad en inglés y en español de los medinfluencers en Instagram. Vivat Academia ► pp. 1 ff. DOI logo
Su, Wenchao, Defeng Li & Jing Ning
2024. Syntactic asymmetry and spillover effects in simultaneous interpreting with slides: an eye-tracking study on beginner interpreters. Perspectives ► pp. 1 ff. DOI logo
Chekin, Leonid S.
2023. First Secretary Gierek, President Carter, and the president’s Polish interpreter. Babel. Revue internationale de la traduction / International Journal of Translation / Revista Internacional de Traducción 69:6 ► pp. 725 ff. DOI logo
Gao, Fei
2023. Getting the message in ‘Sound’ across at conference interpreting: a case study on rendering prosodic emphasis. Perspectives 31:6 ► pp. 1033 ff. DOI logo
Yuan, Lu & Binhua Wang
2023. Cognitive processing of the extra visual layer of live captioning in simultaneous interpreting. Triangulation of eye-tracked process and performance data. Ampersand 11 ► pp. 100131 ff. DOI logo
Yuan, Lu & Binhua Wang
2024. Eye-tracking the processing of visual input of live transcripts in remote simultaneous interpreting. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 22:1 ► pp. 118 ff. DOI logo
Zhao, Nan, Zhenguang G. Cai, Yanping Dong & Daniel Mirman
2023. Speech errors in consecutive interpreting: Effects of language proficiency, working memory, and anxiety. PLOS ONE 18:10 ► pp. e0292718 ff. DOI logo
Cheung, Andrew K. F.
2022. Remote Simultaneous Interpreting from Home or Hub: Accuracy of Numbers from English into Mandarin Chinese. In Translation and Interpreting in the Age of COVID-19 [Corpora and Intercultural Studies, 9], ► pp. 113 ff. DOI logo
Cheung, Andrew K. F.
2024. Cognitive load in remote simultaneous interpreting: place name translation in two Mandarin variants. Humanities and Social Sciences Communications 11:1 DOI logo
Chmiel, Agnieszka & Nicoletta Spinolo
2022. Testing the impact of remote interpreting settings on interpreter experience and performance. Translation, Cognition & Behavior 5:2 ► pp. 250 ff. DOI logo
Zhu, Xuelian & Vahid Aryadoust
2022. A Synthetic Review of Cognitive Load in Distance Interpreting: Toward an Explanatory Model. Frontiers in Psychology 13 DOI logo
Kruger, Jan-Louis
2021. Eye tracking. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 79 ff. DOI logo
Pöchhacker, Franz
2021. Multimodality in interpreting. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 151 ff. DOI logo
Seeber, Kilian G., Laura Keller & Alexis Hervais-Adelman
2020. When the ear leads the eye – the use of text during simultaneous interpretation. Language, Cognition and Neuroscience 35:10 ► pp. 1480 ff. DOI logo
Su, Wenchao
2020. Issues and Approaches to CTIS. In Eye-Tracking Processes and Styles in Sight Translation [New Frontiers in Translation Studies, ], ► pp. 9 ff. DOI logo

This list is based on CrossRef data as of 4 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue