Cover not available

Article published In: Technology
Edited by Franz Pöchhacker and Minhua Liu
[Interpreting 26:2] 2024
► pp. 231–252

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Visual processing during computer-assisted consecutive interpreting

Evidence from eye movements

ORCiD logo with link | Guangdong University of Foreign Studies
ORCiD logo with link | Macquarie University
Published online: 5 July 2024
DOI logo with link
https://doi.org/10.1075/intp.00104.che
Abstract
This study investigates the visual processing patterns during computer-assisted consecutive interpreting (CACI). In phase I of the proposed CACI workflow, the interpreter listens to the source speech and respeaks it into speech recognition (SR) software. In phase II, the interpreter produces target speech supported by the SR text and its machine translation (MT) output. A group of students performed CACI with their eye movements tracked. In phase I, the participants devoted the majority of their attention to listening and respeaking, with very limited attention distributed to the SR text. However, a positive correlation was found between the percentage of dwell time on the SR text and the quality of respeaking, which suggests that active monitoring could be important. In phase II, the participants devoted more visual attention to the MT text than to the SR text and engaged in deeper and more effortful processing when reading the MT text. We identified a positive correlation between the percentage of dwell time on the MT text and interpreting quality in the L2–L1 direction but not in the L1–L2 direction. These results contribute to our understanding of computer-assisted interpreting and can provide insights for future research and training in this area.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Eye movements in reading
    • 2.2Eye movements in respeaking, sight translation and post-editing
  • 3.The present study
    • 3.1Participants
    • 3.2Apparatus
    • 3.3Procedure
    • 3.4Data and analysis
  • 4.Results
    • 4.1Visual processing in the respeaking phase
    • 4.2Visual processing in the production phase
  • 5.Discussion
  • 6.Conclusion
  • References
References (50)
References
Barr, D. J., Levy, R., Scheepers, C. & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 68 (3), 255–278. Google Scholar logo with link to Google Scholar
Bates, D., Mächler, M., Bolker, B. & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software 671, 1–48. Google Scholar logo with link to Google Scholar
Carl, M., Dragsted, B., Elming, J., Hardt, D. & Jakobsen, A. L. (2011). The process of post-editing: A pilot study. In A. L. Jakobsen, M. Carl, B. Sharp & M. Zock (Eds.), Proceedings of the 8th International NLPCS Workshop. Frederiksberg: Samfundslitteratur, 131–142.Google Scholar logo with link to Google Scholar
Chafe, W. (1985). Linguistic differences produced by differences between speaking and writing. In D. R. Olson, A. Hildyard & N. Torrance (Eds.), Literacy, language, and learning: The nature and consequences of reading and writing. Cambridge: Cambridge University Press, 105–122.Google Scholar logo with link to Google Scholar
Chen, S. (2017). Note-taking in consecutive interpreting: New data from pen recording. Translation and Interpreting 9 (1), 4–23. Google Scholar logo with link to Google Scholar
(2020a). The impact of directionality on the process and product in consecutive interpreting between Chinese and English: Evidence from pen recording and eye tracking. The Journal of Specialised Translation 341, 100–117.Google Scholar logo with link to Google Scholar
(under review). A computer-assisted consecutive interpreting mode: Training and evaluation.
(2022). Reading patterns, reformulation and eye-voice span (IVS) in sight translation. Translation and Interpreting Studies 18 (2), 213–234. Google Scholar logo with link to Google Scholar
Cop, U., Drieghe, D. & Duyck, W. (2015). Eye movement patterns in natural reading: A comparison of monolingual and bilingual reading of a novel. PLoS ONE 10 (8), e0134008. Google Scholar logo with link to Google Scholar
Dam, H. V. & Engberg, J. (2006). Assessing accuracy in consecutive interpreting: a comparison of semantic network analyses and intuitive assessments. In C. Heine, K. Schubert & H. Gerzymisch-Arbogast (Eds.), Text and translation: Theory and methodology of translation. Tübingen: Narr Francke Attempto Verlag, 215–234.Google Scholar logo with link to Google Scholar
Davitti, E. & Sandrelli, A. (2020). Embracing the complexity: A pilot study on interlingual respeaking. Journal of Audiovisual Translation 3 (2), 103–139. Google Scholar logo with link to Google Scholar
Demareva, V. & Edeleva, Y. (2020). Eye-tracking based L2 detection: Universal and specific eye movement patterns in L1 and L2 reading. Procedia Computer Science 1691, 673–676. Google Scholar logo with link to Google Scholar
Downie, J. (2023). Where is it all going? Technology, economic pressures and the future of interpreting. In G. Corpas Pastor & B. Defrancq (Eds.), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 277–301. Google Scholar logo with link to Google Scholar
Dragsted, B. & Hansen, I. G. (2009). Exploring translation and interpreting hybrids. The case of sight translation. Meta 54 (3), 588–604. Google Scholar logo with link to Google Scholar
Eskenazi, M. A. (2023). Best practices for cleaning eye movement data in reading research. Behavior Research Methods. Google Scholar logo with link to Google Scholar
Fantinuoli, C. (2023). Towards AI-enhanced computer-assisted interpreting. In G. Corpas Pastor & B. Defrancq (Eds), Interpreting technologies – current and future trends. Amsterdam: John Benjamins, 46–71. Google Scholar logo with link to Google Scholar
Gerver, D. (1969/2002). The effects of source language presentation rate on the performance of simultaneous conference interpreters. In F. Pöchhacker & M. Shlesinger (Eds.), The interpreting studies reader. London and New York: Routledge, 53–66.Google Scholar logo with link to Google Scholar
Han, C. (2018). Latent trait modelling of rater accuracy of formative peer assessment of English-Chinese consecutive interpreting. Assessment & Evaluation in Higher Education 43(6), 979–994. Google Scholar logo with link to Google Scholar
Ho, C-E. (2017). An integrated eye-tracking study into the cognitive process of English–Chinese sight translation: Impacts of training and experience. PhD dissertation, National Taiwan Normal University.
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H. & van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Huang, C-C. (2011). Tracking eye movements in sight translation. MA thesis, National Taiwan Normal University.
Koglin, A. (2015). An empirical investigation of cognitive effort required to post-edit machine translated metaphors compared to the translation of metaphors. Translation and Interpreting 7 (1), 126–141.Google Scholar logo with link to Google Scholar
Kuperman, V., Siegelman, N., Schroeder, S., Acartürk, C., Alexeeva, S., Amenta, S. … Usal, K. A. (2022). Text reading in English as a second language: Evidence from the Multilingual Eye-Movements Corpus. Studies in Second Language Acquisition 45 (1), 3–37. Google Scholar logo with link to Google Scholar
Lijewska, A., Chmiel, A. & Inhoff, A. W. (2022). Stages of sight translation: Evidence from eye movements. Applied Psycholinguistics 431, 997–1018. Google Scholar logo with link to Google Scholar
Moorkens, J., O’Brien, S., da Silva, I. A. L., Fonseca, N. B. d. L. & Alves, F. (2015). Correlations of perceived post-editing effort with measurements of actual effort. Machine Translation 291, 267–284. Google Scholar logo with link to Google Scholar
O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation 25 (3), 197–215. Google Scholar logo with link to Google Scholar
Pan, J., Yan, M. & Laubrock, J. (2017). Perceptual span in oral reading: The case of Chinese. Scientific Studies of Reading 21 (3), 254–263. Google Scholar logo with link to Google Scholar
Parshina, O., Laurinavichyute, A. K. & Sekerina, I. A. (2021). Eye-movement benchmarks in Heritage Language reading. Bilingualism: Language and Cognition 24 (1), 69–82. Google Scholar logo with link to Google Scholar
Pisani, E. & Fantinuoli, C. (2021). Measuring the impact of automatic speech recognition on number rendition in simultaneous interpreting. In C. Wang & B. Zheng (Eds.), Empirical studies of translation and interpreting. New York: Routledge, 181–197. Google Scholar logo with link to Google Scholar
R Core Team (2023). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.Google Scholar logo with link to Google Scholar
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin 124 (3), 372–422. Google Scholar logo with link to Google Scholar
(2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology 62 (8), 1457–1506. Google Scholar logo with link to Google Scholar
Rinne, J. O., Tommola, J., Laine, M., Krause, B. J., Schmidt, D., Kaasinen, V., Teräs, M., Sipilä, H. & Sunnari, M. (2000). The translating brain: Cerebral activation patterns during simultaneous interpreting. Neuroscience Letters 294 (2), 85–88. Google Scholar logo with link to Google Scholar
Romero-Fresco, P. (2011). Subtitling through speech recognition: Respeaking. Manchester: Routledge.Google Scholar logo with link to Google Scholar
(2015). Respeaking. In F. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. Abingdon: Routledge, 350–351.Google Scholar logo with link to Google Scholar
Romero-Fresco, P. & Martínez, J. (2015). Accuracy rate in live subtitling: The NER model. In R. Baños & J. Díaz-Cintas (Eds.), Audiovisual translation in a global context: Mapping an ever-changing landscape. London: Palgrave Macmillan, 28–50. Google Scholar logo with link to Google Scholar
Scott, S. D., Sharpe, H., O’Leary, K., Dehaeck, U., Hindmarsh, K., Moore, J. G. & Osmond, M. H. (2009). Court reporters: A viable solution for the challenges of focus group data collection? Qualitative Health Research 19 (1), 140–146. Google Scholar logo with link to Google Scholar
Shreve, G. M., Lacruz, I. & Angelone, E. (2010). Cognitive effort, syntactic disruption, and visual interference in a sight translation task. In G. M. Shreve & E. Angelone (Eds.). Translation and cognition. Amsterdam and Philadelphia: John Benjamins, 63–84. Google Scholar logo with link to Google Scholar
Sun, F. (1993). Eye movements in reading Chinese: Paragraphs, single characters and pinyin. In S. F. Wright & R. Groner (Eds.), Facets of dyslexia and its remediation. Amsterdam: North-Holland, 245–255. Google Scholar logo with link to Google Scholar
Sun, F., Morita, M. & Stark, L. W. (1985). Comparative patterns of reading eye movement in Chinese and English. Perception and Psychophysics 371, 502–506. Google Scholar logo with link to Google Scholar
Szarkowska, A., Dutka, Ł., Szychowska, A. & Pilipczuk, O. (2018). Visual attention distribution in intralingual respeaking: An eye-tracking study. In C. Walker & F. M. Federici (Eds.), Eye tracking and multidisciplinary studies on translation. Amsterdam: John Benjamins, 185–201. Google Scholar logo with link to Google Scholar
Toral, A., Wieling, M. & Way, A. (2018). Post-editing effort of a novel with statistical and neural machine translation. Frontiers in Digital Humanities 51, 1–11. Google Scholar logo with link to Google Scholar
White, S. J., Warrington, K. L., McGowan, V. A. & Paterson, K. B. (2015). Eye movements during reading and topic scanning: Effects of word frequency. Journal of Experimental Psychology: Human Perception and Performance 41 (1), 233–248.Google Scholar logo with link to Google Scholar
Cited by (2)

Cited by two other publications

Chen, Lei
2025. Examining cognitive load in human-machine collaborative translation: insights from eye-tracking experiments of Chinese-English translation. Frontiers in Psychology 16 DOI logo
Zhang, Wenkang & Rui Xie
2025. Book review: Zhao Tianyuan, «英汉交替传译源语难度评估体系构建研究» [An Empirical Study on Constructing the Difficulty Assessment System of Source Materials for E-C Consecutive Interpreting]. Interpreting and Society DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue