Cover not available

Article published In: Translation, Cognition & Behavior
Vol. 6:1 (2023) ► pp.60–86

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB

The impact of traditional and interactive post-editing on Machine Translation User Experience, quality, and productivity

ORCiD logo with link | Dublin City University
ORCiD logo with link | Dublin City University
ORCiD logo with link | University College Dublin
Open Access logo
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Open Access publication of this article was funded through a Transformative Agreement with Dublin City University.

Published online: 13 June 2023
DOI logo with link
https://doi.org/10.1075/tcb.00077.bri
Abstract
This paper presents a user study with 15 professional translators in the English-Spanish combination. We present the concept of Machine Translation User Experience (MTUX) and compare the effects of traditional post-editing (TPE) and interactive post-editing (IPE) on MTUX, translation quality and productivity. Results suggest that translators prefer IPE to TPE because they are in control of the interaction in this new form of translator-computer interaction and feel more empowered in their interaction with Machine Translation. Productivity results also suggest that IPE may be an interesting alternative to TPE, given the fact that translators worked faster in IPE even though they had no experience in this new machine translation post-editing modality, but were already used to TPE.
Article outline
  • 1.Introduction
  • 2.Traditional vs interactive post-editing
  • 3.Machine Translation User Experience (MTUX)
  • 4.Methodology
    • Participants
    • Texts
    • IPE workbench
    • Design of the controlled study
    • Measures for MTUX, productivity, and quality
      • Machine Translation User Experience (MTUX)
      • Translation productivity
      • Translation quality
  • 5.Results
    • Analysis approach
    • RQ1. Do the MTPE modalities (TPE or IPE) impact MTUX in a statistically significantly way?
    • RQ2. As translators have more experience in TPE than in IPE, is MTUX impacted by increased experience with the system in a specific MTPE modality?
    • RQ3. Do the MTPE modalities (TPE or IPE) impact translator performance measures (translation quality and productivity) in a statistically significantly way?
      • Translation quality
      • Translation productivity
    • RQ4. Does MTUX correlate with translator performance measures (translation quality and productivity)?
  • 6.Discussion
    • IPE showed significant MTUX benefits to TPE
    • IPE and TPE experience over time—The need for a longitudinal study
    • The relationship between MTUX and performance
  • 7.Conclusions
  • Acknowledgements
  • Notes
  • References
References (60)
References
Alabau Gonzalvo, Vicent, Ragnar Bonkb, Christian Buck, Michael Carlb, Francisco Casacuberta Nolla, Mercedes García Martínez, Jesús González Rubio, Philip Koehn, Luis Alberto Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Germán Sanchis Trilles & Chara Tsoukalak. 2013. CASMACAT: An open source workbench for advanced computer aided translation. Prague Bulletin of Mathematical Linguistics, 1001: 101–12. De Gruyter Open. Google Scholar logo with link to Google Scholar
Alabau Gonzalvo, Vicent, Michael Carl, Mercedes García Martínez & Jesús González Rubio. 2016. Learning Advanced Post-Editing. In New Directions in Empirical Translation Process Research. Edited by Michael Carl, Srinivas Bangalore, Moritz Schaeffer, 95–110. Cham: Springer. Google Scholar logo with link to Google Scholar
Alves, Fabio, Karina Sarto Szpak, José Luiz Gonçalves, Kyoko Sekino, Marceli Aquino, Rodrigo Araújo e Castro, Arlene Koglin, Norma B. de Lima Fonseca, and Bartolomé Mesa Lao. 2016. Investigating Cognitive Effort in Post-Editing: A Relevance-Theoretical Approach. In Eyetracking and Applied Linguistics. Edited by Silvia Hansen-Schirra & Sambor Grucza, 109–42. Language Science Press. [URL]
Artstein, Ron. 2017. Inter-Annotator Agreement. In Handbook of Linguistic Annotation. Edited by Nancy Ide & James Pustejovsky, 297–313. Dordrecht: Springer Netherlands. Google Scholar logo with link to Google Scholar
Bertoldi, Nicola, Davide Caroselli & Marcello Federico. 2018. The ModernMT Project. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation. Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 345–347. Alicante: European Association for Machine Translation. [URL]
Bowker, Lynne & Jairo Buitrago Ciro. 2018. Localizing Websites Using Machine Translation: Exploring Connections between User EXperience and Translatability. In The Human Factor in Machine Translation. Edited by Sin-wai Chan, 29–52. London: Routledge. Google Scholar logo with link to Google Scholar
Briva-Iglesias, Vicent. 2021. Traducción humana vs. traducción automática: análisis contrastivo e implicaciones para la aplicación de la traducción automática en traducción jurídica. Mutatis Mutandis. Revista Latinoamericana de Traducción 14 (2): 571–600. Google Scholar logo with link to Google Scholar
. 2022. English-Catalan Neural Machine Translation: State-of-the-Art Technology, Quality, and Productivity. Tradumàtica: Tecnologies de La Traducció 201: 149–76. Google Scholar logo with link to Google Scholar
Briva-Iglesias, Vicent, Sharon O’Brien & Benjamin R. Cowan. 2023. Measuring Machine Translation User Experience: A Comparison between AttrakDiff and User Experience Questionnaire. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation.Google Scholar logo with link to Google Scholar
. Forthcoming. Translators’ Pre-Task Perceptions of CAT Tools and MTPE, and Their Relationship with Translation Quality: Implications for Training.
Cadwell, Patrick, Sheila Castilho, Sharon O’Brien & Linda Mitchell. 2016. Human Factors in Machine Translation and Post-Editing among Institutional Translators. Translation Spaces 5 (2): 222–43. Google Scholar logo with link to Google Scholar
Cadwell, Patrick, Sharon O’Brien & Carlos S. C. Teixeira. 2018. Resistance and Accommodation: Factors for the (Non-) Adoption of Machine Translation among Professional Translators. Perspectives 26 (3): 301–21. Google Scholar logo with link to Google Scholar
Castilho, Sheila. 2016. Measuring Acceptability of Machine Translated Enterprise Content. PhD Thesis, Dublin City University. Doctoral, Dublin City University. Faculty of Humanities and Social Sciences. [URL]
Castilho, Sheila, Stephen Doherty, Federico Gaspari & Joss Moorkens. 2018. Approaches to Human and Machine Translation Quality Assessment. In Translation Quality Assessment: From Principles to Practice. Edited by Joss Moorkens, Sheila Castilho, Federico Gaspari & Stephen Doherty, 9–38. Machine Translation: Technologies and Applications. Cham: Springer International Publishing. Google Scholar logo with link to Google Scholar
Daems, Joke & Lieve Macken. 2019. Interactive Adaptive SMT versus Interactive Adaptive NMT: A User Experience Evaluation. Machine Translation 33 (1): 117–34. Google Scholar logo with link to Google Scholar
Dix, Alan. 2010. Human-Computer Interaction: A Stable Discipline, a Nascent Science, and the Growth of the Long Tail. Interacting with Computers 22 (1): 13–27. Google Scholar logo with link to Google Scholar
Esteban Lauzán, José, José Lorenzo Mon, Antonio Sánchez Valderrábanos & Guy Lapalme. 2004. TransType2: An Innovative Computer-Assisted Translation System. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, 94–97. Barcelona, Spain: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Etchegoyhen, Thierry, Anna Fernández Torné, Andoni Azpeitia Zaldua, Eva Martínez García & Anna Matamala Ripoll. 2018. Evaluating Domain Adaptation for Machine Translation Across Scenarios. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association. [URL]
Forlizzi, Jodi & Katja Battarbee. 2004. Understanding Experience in Interactive Systems. In Proceedings of the 5th Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, 261–68. DIS ’04. New York, NY, USA: Association for Computing Machinery. Google Scholar logo with link to Google Scholar
Gaspari, Federico, Antonio Toral Ruiz, Sudip Kumar Naskar, Declan Groves & Andy Way. 2014. Perception vs Reality: Measuring Machine Translation Post-Editing Productivity. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, 60–72. Vancouver, Canada: Association for Machine Translation in the Americas.Google Scholar logo with link to Google Scholar
Green, Spence. 2016. Interactive Machine Translation. In Conferences of the Association for Machine Translation in the Americas. Invited talk.Google Scholar logo with link to Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer & Christopher D. Manning. 2014. Predictive Translation Memory: A Mixed-Initiative System for Human Language Translation. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, 177–187. Honolulu Hawaii USA: ACM. Google Scholar logo with link to Google Scholar
Green, Spence, Sida I. Wang, Jason Chuang, Jeffrey Heer, Sebastian Schuster & Christopher D. Manning. 2014. Human Effort and Machine Learnability in Computer Aided Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1225–1236. Doha, Qatar: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Guerberof Arenas, Ana, Joss Moorkens & Sharon O’Brien. 2021. The Impact of Translation Modality on User Experience: An Eye-Tracking Study of the Microsoft Word User Interface. Machine Translation 35 (2): 205–37. Google Scholar logo with link to Google Scholar
Guerberof Arenas, Ana. 2008. Productivity and Quality in the Post-Editing of Outputs from Translation Memories and Machine Translation. Localisation Focus The International Journal of Localisation, 7 (1), 11–21.Google Scholar logo with link to Google Scholar
ISO. 2018. ISO 9241-11:2018(En), Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. 2018. [URL]
Karakanta, Alina, Luisa Bentivogli, Mauro Cettolo, Matteo Negri & Marco Turchi. 2022. Post-Editing in Automatic Subtitling: A Subtitlers’ Perspective. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 261–70. Ghent, Belgium: European Association for Machine Translation. [URL]
Knowles, Rebecca, Marina Sánchez Torrón & Philipp Koehn. 2019. Neural Interactive Translation Prediction. Machine Translation 331: 135–134. Google Scholar logo with link to Google Scholar
Koehn, Philipp. 2009. A Web-Based Interactive Computer Aided Translation Tool. In Proceedings of the ACL-IJCNLP 2009 Software Demonstrations. Edited by Gary Geunbae Lee, Sabine Schulte im Walde, 17–20. Suntec, Singapore: Association for Computational Linguistics. [URL].
Koehn, Philipp, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, Evan Herbst, Hieu Hoang, et al 2007. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions—ACL. Edited by Sophia Ananiadou, 177–180. Prague: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Koponen, Maarit. 2012. Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations. In Proceedings of the Seventh Workshop on Statistical Machine Translation. Edited by Chris Callison-Burch, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut & Lucia Specia, 181–190. Montréal, Canada: Association for Computational Linguistics.Google Scholar logo with link to Google Scholar
Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen & Jörg Tiedemann. 2020. MT for Subtitling: Investigating Professional Translators’ User Experience and Feedback. In Proceedings of 1st Workshop on Post-Editing in Modern-Day Translation. Edited by John E. Ortega, Marcello Federico, Constantin Orasan, Maja Popovic, 79–92. Virtual: AMTA. [URL]
Langlais, Philippe, George Foster & Guy Lapalme. 2000. TransType: A Computer-Aided Translation Typing System. In ANLP-NAACL 2000 Workshop: Embedded Machine Translation Systems, 1–6. [URL]
Läubli, Samuel, Sheila Castilho, Graham Neubig, Rico Sennrich, Qinlan Shen & Antonio Toral Ruiz. 2020. A Set of Recommendations for Assessing Human–Machine Parity in Language Translation. Journal of Artificial Intelligence Research 671 (March): 653–672–653–72. Google Scholar logo with link to Google Scholar
Läubli, Samuel & Spence Green. 2019. Translation Technology Research and Human–Computer Interaction. In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 370–83. New York, NY, USA: Routledge. Google Scholar logo with link to Google Scholar
Läubli, Samuel, Patrick Simianer, Joern Wuebker, Geza Kovacs, Rico Sennrich & Spence Green. 2022. The Impact of Text Presentation on Translator Performance. Target. International Journal of Translation Studies 34 (2): 309–342. [URL].
Laugwitz, Bettina, Theo Held & Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. International Journal of Interactive Multimedia and Artificial Intelligence 4 (4): 63–76. Google Scholar logo with link to Google Scholar
Macklovitch, Elliott. 2006. TransType2: The Last Word. In Proceedings of the 5th Edition of the International Conference on Language Resources and Evaluation. Edited by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, Daniel Tapias, 1–6. Genoa, Italy: International Conference on Language Resources and Evaluation (LREC).Google Scholar logo with link to Google Scholar
Martikainen, Hanna. 2022. Ghosts in the Machine: Can Adaptive MT Help Reclaim a Place for the Human in the Loop?. In Portail HAL Sorbonne Nouvelle. [URL]
Nurminen, Mary. 2019. Decision-Making, Risk, and Gist Machine Translation in the Work of Patent Professionals. In Proceedings of the 8th Workshop on Patent and Scientific Literature Translation, 32–42. Dublin: European Association for Machine Translation. [URL]
Nurminen, Mary & Niko Papula. 2018. Gist MT Users: A Snapshot of the Use and Users of One Online MT Tool. In Edited by Juan Antonio Pérez-Ortiz, Felipe Sánchez-Martínez, Miquel Esplà-Gomis, Maja Popovic, Celia Rico, André Martins, Joachim Van den Bogaert, Mikel L. Forcada, 199–208. Alicante: European Association for Machine Translation. [URL]
O’Brien, Sharon. 2006. Pauses as Indicators of Cognitive Effort in Post-Editing Machine Translation Output. Across Languages and Cultures 7 (1): 1–21. Google Scholar logo with link to Google Scholar
. 2022. How to Deal with Errors in Machine Translation: Post-Editing. In Machine Translation for Everyone, 105–20. Berlin: Language Science Press. Google Scholar logo with link to Google Scholar
O’Brien, Sharon, Maureen Ehrensberger-Dow, Marcel Hasler & Megan Connolly. 2017. Irritating CAT Tool Features That Matter to Translators. Hermes: Journal of Language and Communication in Business 561: 145–62.Google Scholar logo with link to Google Scholar
Olohan, Maeve. 2011. Translators and Translation Technology: The Dance of Agency. Translation Studies 4 (3): 342–57. Google Scholar logo with link to Google Scholar
Paz, Freddy & Jose Pow-Sang. 2016. A Systematic Mapping Review of Usability Evaluation Methods for Software Development Process 101: 165–78. Google Scholar logo with link to Google Scholar
Pérez Macías, Lorena, María del Mar Sánchez Ramos & Celia Rico Pérez. 2020. Study on the Usefulness of Machine Translation in the Migratory Context: Analysis of Translators’ Perceptions. Open Linguistics 6 (1): 68–76. Google Scholar logo with link to Google Scholar
Peris Abril, Álvaro, Miguel Domingo Ballester & Francisco Casacuberta Nolla. 2017. Interactive Neural Machine Translation. Computer Speech & Language 451 (September): 201–20. Google Scholar logo with link to Google Scholar
Sadiku, Matthew N. O. & Sarhan M. Musa. 2021. A Primer on Multiple Intelligences. Cham: Springer International Publishing. Google Scholar logo with link to Google Scholar
Sakamoto, Akiko. 2019. Unintended Consequences of Translation Technologies: From Project Managers Perspectives’. Perspectives 27 (1): 58–73. Google Scholar logo with link to Google Scholar
Sánchez-Torrón, Marina. 2017. Productivity in Post-Editing and in Neural Interactive Translation Prediction: A Study of English-to-Spanish Professional Translators. PhD Dissertation. University of Auckland. [URL]
Sánchez Gijón, Pilar, Joss Moorkens & Andy Way. 2019. Post-Editing Neural Machine Translation versus Translation Memory Segments. Machine Translation 33 (1–2): 31–59. Google Scholar logo with link to Google Scholar
Sanchis Trilles, Germán, Vicent Alabau Gonzalvo, Christian Buck, Michael Carl, Francisco Casacuberta Nolla, Mercedes García Martínez, Ulrich Germann, Jesús González Rubio, Robin L. Hill, Philipp Koehn, Luis A. Leiva Torres, Bartolomé Mesa Lao, Daniel Ortiz Martínez, Herve Saint-Amand, Chara Tsoukala & Enrique Vidal Ruiz. 2014. Interactive Translation Prediction versus Conventional Post-Editing in Practice: A Study with the CasMaCat Workbench, Machine Translation 28 (3): 217–235. Google Scholar logo with link to Google Scholar
Schrepp, Martin, Andreas Hinderks & Jörg Thomaschewski. 2014. Applying the User Experience Questionnaire (UEQ) in Different Evaluation Scenarios. In Design, User Experience, and Usability. Theories, Methods, and Tools for Designing the User Experience, edited by Aaron Marcus, 383–392. Lecture Notes in Computer Science. Cham: Springer. Google Scholar logo with link to Google Scholar
Torregrosa Rivero, Daniel. 2018. Black-box interactive translation prediction. PhD dissertation, Universidad de Alicante. [URL]
Vermeeren, Arnold P. O. S., Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout & Kaisa Väänänen-Vainio-Mattila. 2010. User Experience Evaluation Methods: Current State and Development Needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, 521–30. NordiCHI ’10. New York, NY, USA: Association for Computing Machinery. Google Scholar logo with link to Google Scholar
Vieira, Lucas Nunes. 2019. Post-Editing of Machine Translation. In The Routledge Handbook of Translation and Technology. Edited by Minako O’Hagan, 319–335. Routledge. Google Scholar logo with link to Google Scholar
Wang, Xiangling, Tingting Wang, Ricardo Muñoz Martín & Yanfang Jia. 2021. Investigating Usability in Postediting Neural Machine Translation: Evidence from Translation Trainees’ Self-Perception and Performance. Across Languages and Cultures 22 (1): 100–123. Google Scholar logo with link to Google Scholar
Weinberg, Bruce A. 2004. ‘Experience and Technology Adoption’. SSRN Scholarly Paper. Rochester, NY. Google Scholar logo with link to Google Scholar
Cited by (12)

Cited by 12 other publications

Amirdabbaghian, Amin
2025. Latest Developments in Translation and Interpreting Technologies in Malaysia. In Cutting-Edge Approaches in Translator Education and Pedagogy, ► pp. 191 ff. DOI logo
He, Shuping, Noor Aireen Ibrahim & Myoung Sook Kang
2025. Website Language Accessibility Checklist. International Journal of E-Health and Medical Communications 16:1 ► pp. 1 ff. DOI logo
Jiang, Liping, Bingbing Wei & Ghayth Kamel Shaker Al-Shaibani
2025. Effective Neural Machine Translation with Human Post-Editing of Chinese Intangible Cultural Heritage Corpus into English. Sage Open 15:4 DOI logo
Lei, Tingting & Jeroen van de Weijer
2025. Evaluating different translation methods: a case study of Chinese graduate students in an MTI program. Cogent Arts & Humanities 12:1 DOI logo
Nurminen, Mary
2025. Is machine translation gisting translation?. Translation Spaces DOI logo
Su, Tong
2025. Cross-Language Translation Evaluation: A Comparative Analysis of the Quality of English-Chinese Interpretation by AI Models and Strategies for Improvement. Literature Language and Cultural Studies 2:1 ► pp. 32 ff. DOI logo
Wu, Liting
2025. Proceedings of the 2nd Guangdong-Hong Kong-Macao Greater Bay Area International Conference on Digital Economy and Artificial Intelligence, ► pp. 1098 ff. DOI logo
Yao, Yao, Tianyi Han & Dechao Li
2025. Measuring translation trainees’ effort in AI-assisted post-editing: a multi-method approach. The Interpreter and Translator Trainer 19:3-4 ► pp. 357 ff. DOI logo
Ciobanu, Dragoș , Miguel Rios, Alina Secară, Justus Brockmann, Raluca-Maria Chereji & Claudia Wiesinger
2024. The impact of using text-to-speech (TTS) in post-editing machine translation (PEMT) workflows on translators’ cognitive effort, productivity, quality, and perceptions. Tradumàtica tecnologies de la traducció :22 ► pp. 323 ff. DOI logo
Daems, Joke
2024. Students’ Attitudes Towards Interactive and Adaptive Translation Technology: Four years of Working with Lilt. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 239 ff. DOI logo
Hao, Yu, Ke Hu & Anthony Pym
2024. Who’s afraid of literary post-editing? Performances and Reflections of Student Translators. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 263 ff. DOI logo
Yang, Yanxia
2024. Understanding machine translation fit for language learning: The mediating effect of machine translation literacy. Education and Information Technologies 29:15 ► pp. 20163 ff. DOI logo

This list is based on CrossRef data as of 5 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue