Cover not available

In:Eye Tracking and Multidisciplinary Studies on Translation
Edited by Callum Walker and Federico M. Federici
[Benjamins Translation Library 143] 2018
► pp. 55–69

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Chapter 4
Eye tracking as a measure of cognitive effort for post-editing of machine translation

ORCiD logo with link
Published online: 16 October 2018
DOI logo with link
https://doi.org/10.1075/btl.143.04moo
Abstract
The three measurements for post-editing effort as proposed by Krings (2001) have been adopted by many researchers in subsequent studies and publications. These measurements comprise temporal effort (the speed or productivity rate of post-editing, often measured in words per second at the segment level), technical effort (the number of actual edits performed by the post-editor, sometimes approximated using the Translation Edit Rate metric (Snover et al. 2006), again usually at the segment level), and cognitive effort. Cognitive effort has been measured using think-aloud protocols, pause measurement, and, increasingly, eye-tracking. This chapter provides a review of studies of post-editing effort using eye-tracking, noting the influence of publications by Danks et al. (1997), and O’Brien (2006, 2008), before describing a single study in detail. The detailed study examines whether predicted affort indicators affect post-editing effort and results were previously published as Moorkens et al. (2015). This chapter focuses instead on methodology and the logistics of running an eye-tracking study recording over 70 sessions. Most of the eye-tracking data analysed were unused in the previous publication, and the small amount presented was not described in detail due to space constraints. In this study average fixation count per segment correlates very strongly with temporal effort, and average fixation duration correlates strongly with technical effort, a result that we compare with other studies of post-editing effort.
Article outline
  • 1.Introduction
  • 2.The task of post-editing
  • 3.Eye-tracking measures used in studies of post-editing
  • 4.Post-editing study
    • 4.1Methodology
    • 4.2Results: Stage 1
    • 4.3Results: Stage 2
    • 4.4Results: Stage 3
  • 5.Conclusion
  • References
References (56)
References
Allen, Jeffrey. 2003. “Post-editing.” In Computers and Translation, ed. by Harold Somers, 297–318. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
ALPAC. 1966. Languages and machines: computers in translation and linguistics. A report by the Automatic Language Processing Advisory Committee. Washington, DC.Google Scholar logo with link to Google Scholar
Alves, Fabio, Arlene Koglin, Bartolomé Mesa-Lao, Mercedes García Martínez, Norma B. de Lima Fonseca, Arthur de Melo Sá, José Luiz Gonçalves, Karina Sarto Szpak, Kyoko Sekino, and Marceli Aquino. 2016. “Analysing the Impact of Interactive Machine Translation on Post-editing Effort.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.Google Scholar logo with link to Google Scholar
Bentivogli, Luisa, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. “Neural versus Phrase-Based Machine Translation Quality: a Case Study.” In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP). Austin, Texas.Google Scholar logo with link to Google Scholar
Beyer, Robert T. 1965. “Hurdling the Language Barrier.” Physics Today 18 (1): 46–52.Google Scholar logo with link to Google Scholar
Bruckner, Christine, and Mirko Plitt. 2001. “Evaluating the Operational Benefit of Using Machine Translation Output as Translation Memory Input.” In Proceedings of MT Summit VIII, 61–65.Google Scholar logo with link to Google Scholar
Caffrey, Colm. 2009. Relevant Abuse? Investigating the Effects of an Abusive Subtitling Procedure on the Perception of TV Anime Using Eye Tracker and Questionnaire. PhD dissertation, Dublin City University.Google Scholar logo with link to Google Scholar
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt Lykke Jakobsen. 2011. “The Process of Post-Editing: a Pilot Study.” In Proceedings of NLPSC 2011.Google Scholar logo with link to Google Scholar
Carl, Michael, Silke Gutermuth, and Silvia Hansen-Schirra. 2015. “Post-editing machine translation: Efficiency, strategies, and revision processes in professional translation settings. Psycholinguistic and Cognitive Inquiries into Translation and Interpreting.” In Psycholinguistic and Cognitive Inquiries into Translation and Interpreting, ed. by Aline Ferreira, and John W. Schwieter, 145–174. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Castilho, Sheila, Joss Moorkens, Federico Gaspari, Iacer Calixto, John Tinsley, and Andy Way. 2017. “Is Neural Machine Translation the New State of the Art?” Prague Bulletin of Mathematical Linguistics 108 (1): 109–120.Google Scholar logo with link to Google Scholar
Chang, Vincent Chieh-Ying. 2009. Testing Applicability of Eye-tracking and fMRI to Translation and Interpreting Studies: An Investigation into Directionality. PhD dissertation. Imperial College, London.Google Scholar logo with link to Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken. 2017. “Identifying the Machine Translation Error Types with the Greatest Impact on Post-Editing Effort.” Frontiers in Psychology 8: 1282.Google Scholar logo with link to Google Scholar
Danks, Joseph H., Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath. 1997. Cognitive Processes in Translation and Interpreting. Thousand Oaks, California: Sage Publications.Google Scholar logo with link to Google Scholar
Denkowski, Michael, and Alon Lavie. 2014 “Meteor Universal: Language Specific Translation Evaluation for Any Target Language.” In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation.Google Scholar logo with link to Google Scholar
Dragsted, Barbara. 2010. “Coordination of Reading and Writing Processes in Translation: An Eye on Uncharted Territory.” In Translation and Cognition, ed. by Gregory M. Schreve, and Erik M. Angelone, 41–62. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Duchowski, Andrew. 2003. Eye Tracking Methodology: Theory and Practice. Heidelberg: Springer.Google Scholar logo with link to Google Scholar
García, Ignacio. 2011. “Translating by Post-editing: Is It the Way Forward?” Machine Translation 25: 217–237.Google Scholar logo with link to Google Scholar
Gaspari, Federico, Antonio Toral, Sudip Kumar Naskar, Declan Groves, Andy Way. 2014. “Perception vs Reality: Measuring Machine Translation Post-Editing Productivity.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 60–72.Google Scholar logo with link to Google Scholar
Gonçalves, José Luiz. 2016. “Investigating Saccades as an Index of Cognitive Effort in Post-editing and Translation.” In Proceedings of EST Congress 2016, Aarhus, Denmark.Google Scholar logo with link to Google Scholar
Hansen-Schirra, Silvia. 2017. “EEG and Universal Language Processing in Translation.” In The Handbook of Translation and Cognition, ed. by John W. Schwieter, and Aline Ferreira, 232–247. Hoboken: John Wiley & Sons.Google Scholar logo with link to Google Scholar
Hokamp, Chris, and Qun Liu. 2015. “HandyCAT.” In Proceedings of European Association for Machine Translation (EAMT) 2015, Antalya, 216.Google Scholar logo with link to Google Scholar
Hutchins, William John. 1992. “Météo.” In An Introduction to Machine Translation, ed. by W. John Hutchins, and Harold L. Somers, 207–220. London: Academic Press.Google Scholar logo with link to Google Scholar
Hvelplund, Kristian T. 2011. Allocation of Cognitive Resources in Translation: An Eye-tracking and Key-logging Study. PhD dissertation, Copenhagen Business School.Google Scholar logo with link to Google Scholar
Jakobsen, Arnt Lykke. 1999. “Logging Target Text Production with Translog.” Copenhagen Studies in Language 24: 9–20.Google Scholar logo with link to Google Scholar
Koglin, Arlene. 2015. “An Empirical Investigation of Cognitive Effort Required to Post-edit Machine Translated Metaphors Compared to the Translation of Metaphors.” Translation and Interpreting 7 (1): 126–141.Google Scholar logo with link to Google Scholar
Krings, Hans P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes. Translated by G. S. Koby. Kent, OH: The Kent State University Press.Google Scholar logo with link to Google Scholar
Lacruz, Isabel, and Gregory M. Shreve. 2014. “Pauses and Cognitive Effort in Post-editing.” In Post-editing of Machine Translation: Processes and Applications, ed. by Sharon O’Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 246–272. Newcastle-Upon-Tyne: Cambridge Scholars.Google Scholar logo with link to Google Scholar
Läubli, Samuel, Mark Fishel, Gary Massey, Maureen Ehrensberger-Dow, and Martin Volk. 2013. “Assessing Post-Editing Efficiency in a Realistic Translation Environment.” In Proceedings of MT Summit XIV Workshop on Post-editing Technology and Practice, Nice, 83–91.Google Scholar logo with link to Google Scholar
Läubli, Samuel, and Ulrich Germann. 2016. “Statistical Modelling and Automatic Tagging of Human Translation Processes.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.Google Scholar logo with link to Google Scholar
Lommel, Arle R., and Donald A. DePalma. 2016. Europe’s Leading Role in Machine Translation: How Europe Is Driving the Shift to MT. Boston: Common Sense Advisory Report.Google Scholar logo with link to Google Scholar
Moorkens, Joss, Sharon O’Brien, Igor Antonio Lourenco Silva, Norma Fonseca, and Fabio Alves. 2015. “Correlations of Perceived Post-editing Effort with Measurements of Actual Effort.” Machine Translation 29 (3–4): 267–284.Google Scholar logo with link to Google Scholar
Moorkens, Joss. 2017. “Under Pressure: Translation in Times of Austerity.” Perspectives: Studies in Translation Theory and Practice, 25 (3): 464–477.Google Scholar logo with link to Google Scholar
Moorkens, Joss, and Sharon O’Brien. 2015. “Post-editing Evaluations: Trade-offs Between Novice and Professional Participants.” In Proceedings of European Association for Machine Translation Conference (EAMT) 2015, Antalya, Turkey, 75–81.Google Scholar logo with link to Google Scholar
Mossop, Brian. 2007. Revising and Editing for Translators. Manchester: St. Jerome.Google Scholar logo with link to Google Scholar
Nitzke, Jean, Katharina Oster. 2016. “Comparing Translation and Post-editing: An Annotation Schema for Activity Units.” In New Directions in Empirical Translation Process Research, ed. by Michael Carl, Srinivas Bangalore, and Moritz Schaeffer, 77–94. Heidelberg: Springer.Google Scholar logo with link to Google Scholar
Nunes Vieira, Lucas. 2015. Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols. PhD dissertation. Newcastle University.Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2005. “Methodologies for Measuring the Correlations Between Post-editing Effort and Machine Translatability.” Machine Translation 19 (1): 37–58.Google Scholar logo with link to Google Scholar
. 2006. “Eye Tracking and Translation Memory Matches.” Perspectives: Studies in Translatology 14 (3): 185–205.Google Scholar logo with link to Google Scholar
. 2008. “Processing Fuzzy Matches in Translation Memory Tools: An Eye-tracking Analysis.” In Looking at Eyes. Eye Tracking Studies of Reading and Translation Processing, ed. by Susanne Göpferich, Arnt Lykke Jakobsen and Inger Mees, 79–102. Frederiksberg: Samfundslitteratur.Google Scholar logo with link to Google Scholar
. 2010. “Introduction to Post-editing: Who, What, How and Where to Next?” In Proceedings of AMTA 2010, the 9th Conference of the Association for Machine Translation in the Americas, Denver, USA.Google Scholar logo with link to Google Scholar
. 2011. “Towards Predicting Post-Editing Productivity”. Machine Translation 25 (3): 197.Google Scholar logo with link to Google Scholar
Plitt, Mirko, and François Masselot. 2010. “A Productivity Test of Statistical Machine Translation Post-editing in a Typical Localization Context.” Prague Bulletin of Mathematical Linguistics 93: 7–16.Google Scholar logo with link to Google Scholar
Saldanha, Gabriela, and Sharon O’Brien. 2014. Research Methodologies in Translation Studies. London: Routledge.Google Scholar logo with link to Google Scholar
Schmidtke, Dag. 2016. “Large Scale Machine Translation Publishing, With Acceptable Quality, for Microsoft Support Content.” In Proceedings of AMTA 2016 Workshop on Interacting with Machine Translation (iMT 2016). Austin, Texas.Google Scholar logo with link to Google Scholar
Senez, Dorothy. 1998. “Post-editing Service for Machine Translation Users at the European Commission.” Translating and the Computer 20. Proceedings of the Twentieth International Conference. 12–13 November 1998.Google Scholar logo with link to Google Scholar
Shreve, Gregory M., and Bruce Diamond. 1997. “Cognitive Processes in Translation and Interpreting: Critical Issues.” In Cognitive Processes in Translation and Interpreting, ed. by Joseph H. Danks, Gregory M. Shreve, Stephen B. Fountain, and Michael K. McBeath, 233–251. Thousand Oaks, California: Sage Publications.Google Scholar logo with link to Google Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of Association for Machine Translation in the Americas, 223–231. 8–12 August 2006, Cambridge, Massachusetts, USA.Google Scholar logo with link to Google Scholar
Specia, Lucia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini. 2009. “Estimating the Sentence-level Quality of Machine Translation Systems.” In Proceedings of the 13th Annual Conference of the EAMT, Barcelona, 28–35.Google Scholar logo with link to Google Scholar
Specia, Lucia. 2011. “Exploiting Objective Annotations for Measuring Translation Post-editing Effort.” In Proceedings of the 15th conference of EAMT, Leuven, 73–80.Google Scholar logo with link to Google Scholar
Su, Keh-Yih, Ming-Wen Wu, and Jing-Shin Chang. 1992. “A New Quantitative Quality Measure for Machine Translation Systems.” In Proceedings of COLING-92, Nantes, 433–439.Google Scholar logo with link to Google Scholar
Teixeira, Carlos S. C. 2014. “Perceived vs. Measured Performance in the Post-editing of Suggestions from Machine Translation and Translation Memories.” In Proceedings of AMTA 2014 Workshop on Post-editing Technology and Practice, Vancouver, 45–59.Google Scholar logo with link to Google Scholar
Turian, Joseph P., Luke Shen, and I. Dan Melamed. 2003. “Evaluation of Machine Translation and Its Evaluation.” In Proceedings of MT Summit 2003, 386–393. New Orleans, Louisiana.Google Scholar logo with link to Google Scholar
Vasconcellos, Muriel, and Marjorie León. 1985. “SPANAM and ENGSPAN: Machine Translation at the Pan American Health Organization.” Computational Linguistics 11 (2–3): 122–136.Google Scholar logo with link to Google Scholar
Wagner, Emma 1985. “Post-editing Systran: A Challenge for Commission Translators.” Terminologie et Traduction 3: 1–7.Google Scholar logo with link to Google Scholar
Way, Andy. 2013. “Traditional and Emerging Use-cases for Machine Translation.” In Proceedings of Translating and the Computer 35.Google Scholar logo with link to Google Scholar
Wilms, Franz-Josef M. 1981. “Von SUSY zu SUSY-BSA: Forderungen and ein anwenderbezogenes MU-System.” Sprache und Datenverarbeitung 5: 38–43.Google Scholar logo with link to Google Scholar
Cited by (15)

Cited by 15 other publications

Yao, Yao, Tianyi Han & Dechao Li
2025. Measuring translation trainees’ effort in AI-assisted post-editing: a multi-method approach. The Interpreter and Translator Trainer 19:3-4 ► pp. 357 ff. DOI logo
Ciobanu, Dragoș , Miguel Rios, Alina Secară, Justus Brockmann, Raluca-Maria Chereji & Claudia Wiesinger
2024. The impact of using text-to-speech (TTS) in post-editing machine translation (PEMT) workflows on translators’ cognitive effort, productivity, quality, and perceptions. Tradumàtica tecnologies de la traducció :22 ► pp. 323 ff. DOI logo
ROJO LÓPEZ, ANA MARÍA, María Inmaculada Vicente López & Kristian Tangsgaard Hvelplund
2024. Measuring cognitive effort in post-editing: an eye-tracking study comparing professional and student translators. Tradumàtica tecnologies de la traducció :22 ► pp. 112 ff. DOI logo
Toledo-Báez, Cristina
2024. Posedición y paridad humano-máquina en traducción automática neuronal: Un estudio empírico desde la traducción profesional. Lebende Sprachen 69:2 ► pp. 434 ff. DOI logo
Tosun, Sümeyra
2024. Machine translation: Turkish–English bilingual speakers’ accuracy detection of evidentiality and preference of MT. Cognitive Research: Principles and Implications 9:1 DOI logo
Cui, Ying, Xiao Liu & Yuqin Cheng
2023. A Comparative Study on the Effort of Human Translation and Post-Editing in Relation to Text Types: An Eye-Tracking and Key-Logging Experiment. Sage Open 13:1 DOI logo
Kasperė, Ramunė, Jurgita Motiejūnienė, Irena Patasienė, Martynas Patašius & Jolita Horbačauskienė
2023. Is machine translation a dim technology for its users? An eye tracking study. Frontiers in Psychology 14 DOI logo
Zou, Longhui, Michael Carl & Devin Gilbert
2023. Integrating Trados-Qualitivity Data to the CRITT TPR-DB: Measuring Post-editing Process Data in an Ecologically Valid Setting. In Corpora and Translation Education [New Frontiers in Translation Studies, ], ► pp. 63 ff. DOI logo
Cumbreño, Cristina & Nora Aranberri
2021. What Do You Say? Comparison of Metrics for Post-editing Effort. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 57 ff. DOI logo
Kruger, Jan-Louis
2021. Eye tracking. In Handbook of Translation Studies [Handbook of Translation Studies, 5], ► pp. 79 ff. DOI logo
Tardel, Anke
2021. Measuring Effort in Subprocesses of Subtitling. In Explorations in Empirical Translation Process Research [Machine Translation: Technologies and Applications, 3], ► pp. 81 ff. DOI logo
Walker, Callum
2021. The Cognitive Paradigm in Translation Studies. In An Eye-Tracking Study of Equivalent Effect in Translation, ► pp. 13 ff. DOI logo
Jiménez-Crespo, Miguel A.
2020. The “technological turn” in translation studies. Translation Spaces 9:2 ► pp. 314 ff. DOI logo
Sánchez-Gijón, Pilar, Joss Moorkens & Andy Way
2019. Post-editing neural machine translation versus translation memory segments. Machine Translation 33:1-2 ► pp. 31 ff. DOI logo
Vardaro, Jennifer, Moritz Schaeffer & Silvia Hansen-Schirra
2019. Translation Quality and Error Recognition in Professional Neural Machine Translation Post-Editing. Informatics 6:3 ► pp. 41 ff. DOI logo

This list is based on CrossRef data as of 3 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue