Cover not available

In:Translation in Transition: Between cognition, computing and technology
Edited by Arnt Lykke Jakobsen and Bartolomé Mesa-Lao
[Benjamins Translation Library 133] 2017
► pp. 161–186

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Chapter 6
From process to product

Links between post-editing effort and post-edited quality

 | University of Bristol
Published online: 30 September 2017
DOI logo with link
https://doi.org/10.1075/btl.133.06vie
Post-editing of machine translation (MT) is now increasingly implemented in the human translation workflow after studies in both industry and academia have demonstrated the efficacy of this practice. Post-editing still involves open questions, however, such as how best to train post-editors and how to estimate the effort required by post-editing tasks. In attempting to address some of these questions, many previous studies investigate the post-editing process, but less research has focused on the post-edited product. This chapter examines the link between the process and product of post-editing by checking to see how post-editing effort data relates to the quality of post-edited texts, assessed in terms of fluency (linguistic quality) and adequacy (translation accuracy). A statistical analysis indicated that the association between editing operations and the fluency of post-edited texts is dependent on the quality of the raw MT output. Interestingly, a negative association was observed between the number of eye fixations on the text and the quality of the post-edited translations. The chapter shows empirical evidence supporting the distinction between the concepts of translation fluency and adequacy, and postulates that automatic processes play a central role in post-editing performance.
Article outline
  • 1.Introduction
  • 2.Related work
  • 3.Methods and procedure
    • 3.1Study materials and the post-editing task
    • 3.2Post-editors
    • 3.3Assessing post-edited quality
    • 3.4Post-editing effort data
  • 4.Results and discussion
    • 4.1Fluency and adequacy
    • 4.2Rater agreement
    • 4.3Post-editing effort and post-edited quality
  • 5.Conclusion
    • 5.1Research findings
    • 5.2Suggestions for future research
  • Acknowledgments
  • Notes
  • References
References (42)
References
Aziz, Wilker, Sheila Castilho, and Lucia Specia. 2012. “PET: a Tool for Post-Editing and Assessing Machine Translation.” In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), 21–27 May 2012, Istanbul, Turkey. ed. by Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Jan Odijk, and Stelios Piperidis, 3982–3987.
Baayen, R. Harald. 2008. Analysing Linguistic Data: a Practical Introduction to Statistics. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. “(Meta-) Evaluation of Machine Translation.” In Proceedings of the Second Workshop on Statistical Machine Translation , Prague, Czech Republic, 23 June 2007, 136–158.
Carl, Michael, Barbara Dragsted, Jakob Elming, Daniel Hardt, and Arnt L. Jakobsen. 2011. “The Process of Post-Editing: a Pilot Study.” In Proceedings of the 8th International NLPCS Workshop: Human-Machine Interaction in Translation, ed. by Bernardette Sharp, Michael Zock, Michael Carl, and Arnt L. Jakobsen, 131–142. Copenhagen: Samfundslitteratur.Google Scholar logo with link to Google Scholar
Christensen, Rune Haubo B. 2010. ordinal: Regression Models for Ordinal Data . R Package Version no. 22.Google Scholar logo with link to Google Scholar
Daems, Joke, Lieve Macken, and Sonia Vandepitte. 2013. “Quality as the Sum of its Parts: a Two-Step Approach for the Identification of Translation Problems and Translation Quality Assessment for HT and MT+PE.” In Proceedings of the MT Summit XIV Workshop on Post-Editing Technology and Practice (WPTP2), 2–6 September 2013, Nice, France, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 63–71.
de Almeida, Giselle. 2013. Translating the Post-Editor: an Investigation of Post-Editing Changes and Correlations with Professional Experience across Two Romance Languages . PhD Thesis, Dublin City University.
de Groot, Annette M.B. 2000. “A Complex-Skill Approach to Translation and Interpreting.” In Tapping and Mapping the Processes of Translation and Interpreting: Outlooks on Empirical Research, ed. by Sonja Tirkkonen-Condit, and Riitta Jääskeläinen, 53–68. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Denkowski, Michael, and Alon Lavie. 2011. “Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems.” In Proceedings of the Sixth Workshop on Statistical Machine Translation (EMNLP 2011), 30–31 July 2011, Edinburgh, UK, 85–91.
Depraetere, Ilse, Nathalie De Sutter, and Arda Tezcan. 2014. “Post-Edited Quality, Post-Editing Behaviour and Human Evaluation: a Case Study.” In Post-Editing of Machine Translation: Processes and Applications, ed. by Sharon O'Brien, Laura Winther Balling, Michael Carl, Michel Simard, and Lucia Specia, 78–108. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar logo with link to Google Scholar
Duchowski, Andrew T. 2007. Eye Tracking Methodology: Theory and Practice. London: Springer-Verlag.Google Scholar logo with link to Google Scholar
Green, Spence, Jason Chuang, Jeffrey Heer, and Christopher D. Manning. 2014. “Predictive Translation Memory: a Mixed-Initiative System for Human Language Translation.” In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology , 05–08 October 2014, Honolulu, USA, 177–187.
Green, Spence, Jeffrey Heer, and Christopher D. Manning. 2013. “The Efficacy of Human Post-Editing for Language Translation.” In ACM Human Factors in Computing Systems (CHI), 27 April–2 May 2013, Paris, France, 439–448.
Gopher, Daniel, and Emanuel Donchin. 1986. “Workload: an Examination of the Concept.” In Handbook of Perception and Human Performance, Vol. 2, ed. by Kenneth R. Boff, Lloyd Kaufman, and James P. Thomas, 1–49. New York: John Wiley and Sons.Google Scholar logo with link to Google Scholar
Guerberof, Ana. 2014. “The Role of Professional Experience in Post-Editing from a Quality and Productivity Perspective.” In Post-Editing of Machine Translation: Processes and Applications, ed. by Sharon O'Brien, Laura Winther Balling, Michael Carl, Michel Simard and Lucia Specia, 51–76. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar logo with link to Google Scholar
Guzmán, Rafael. 2007. “Manual MT Post-Editing: If It’s Not Broken, Don’t Fix It!” Translation Journal 11 (4). Accessed 26 June 2014. [URL].Google Scholar logo with link to Google Scholar
Hogarth, Robin M. 2005. “Deciding Analytically or Trusting Your Intuition? The Advantages and Disadvantages of Analytic and Intuitive Thought.” In The Routines of Decision Making, ed. by Tilmann Betsch, and Susanne Haberstroh, 67–82. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Hvelplund, Kristian Tangsgaard. 2011. Allocation of Cognitive Resources in Translation: An Eye-Tracking and Key-Logging Study . PhD Thesis, Copenhagen Business School.
Jensen, K.T. Hvelplund, Annette Sjørup, and Laura Winther Balling. 2009. “Effects of L1 Syntax on L2 Translation.” In Methodology, Technology and Innovation in Translation Process Research, ed. by Inger M. Mees, Fabio Alves, and Susanne Göpferich, 319–339. Copenhagen: Samfundslitteratur.Google Scholar logo with link to Google Scholar
Koponen, Maarit. 2012. “Comparing Human Perceptions of Post-Editing Effort with Post-Editing Operations.” In Proceedings of the Seventh Workshop on Statistical Machine Translation (NAACL 2012), 7–8 June 2012, Montreal, Canada, 181–190.
Koponen, Maarit, Wilker Aziz, Luciana Ramos, and Lucia Specia. 2012. “Post-Editing Time as a Measure of Cognitive Effort.” In Proceedings of the AMTA 2012 Workshop on Post-Editing Technology and Practice (WPTP 2012) , 28 October 2012, San Diego, USA, ed. by Sharon O'Brien, Michel Simard, and Lucia Specia, 10.
Krings, Hans P. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-Editing Processes, ed. by Geoffrey Koby. Kent, Ohio: Kent State University Press.Google Scholar logo with link to Google Scholar
LDC. 2005. Linguistic Data Annotation Specification: Assessment of Fluency and Adequacy in Translations. Revision 1.5.Google Scholar logo with link to Google Scholar
Lin, Chin-Yew, and Franz Josef Och. 2004. “ORANGE: a Method for Evaluating Automatic Evaluation Metrics for Machine Translation.” In Proceedings of the 20th International Conference on Computational Linguistics , 23–27 August 2004, Geneva, Switzerland, 7.
McCutchen, Deborah. 1996. “A Capacity Theory of Writing: Working Memory in Composition.” Educational Psychology Review 8:299–325. .Google Scholar logo with link to Google Scholar
Meara, Paul, and Barbara Buxton. 1987. “An Alternative to Multiple Choice Vocabulary Tests.” Language Testing 4 (2): 142–154. Google Scholar logo with link to Google Scholar
Mitchell, Linda. 2015. Community Post-Editing of Machine-Translated User-Generated Content . PhD Thesis, Dublin City University.
Norman, Donald A., and Tim Shallice. 1986. “Attention to Action: Willed and Automatic Control of Behaviour.” In Consciousness and Self-Regulation: Advances in Research and Theory, ed. by Richard J. Davidson, Gary E. Schwarts, and David Shapiro, 1–18. New York: Plenum Press. Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2011. “Towards Predicting Post-Editing Productivity.” Machine Translation 25 (3): 197–215. .Google Scholar logo with link to Google Scholar
Paas, Fred G. 1992. “Training Strategies for Attaining Transfer of Problem-Solving Skill in Statistics: a Cognitive-Load Approach.” Journal of Educational Psychology 84 (4): 429–434. Google Scholar logo with link to Google Scholar
Plitt, Mirko, and François Masselot. 2010. “A Productivity Test of Statistical Machine Translation Post-Editing in a Typical Localization Context.” Prague Bull Math Linguist 93: 7–16. Google Scholar logo with link to Google Scholar
Rayner, Keith. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124 (3): 372–422. Google Scholar logo with link to Google Scholar
Snijders, Tom A.B., and Roel Bosker. 1999. Multilevel Analysis: an Introduction to Basic and Advanced Multilevel Modelling. London: Sage.Google Scholar logo with link to Google Scholar
Snover, Matthew, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of the 7th Conference of the Association for Machine Translation of the Americas – Visions for the Future of Machine Translation , 8–12 August 2006, Cambridge, USA, 223–231.
TAUS. 2013. Adequacy/Fluency Guidelines. Accessed 17 June 2015. [URL].Google Scholar logo with link to Google Scholar
TAUS/CNGL. 2010. MT Post-Editing Guidelines. Accessed 17 June 2015. [URL].Google Scholar logo with link to Google Scholar
Temnikova, Irina. 2010. “A Cognitive Evaluation Approach for a Controlled Language Post-Editing Experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010) , 19–21 May 2010, Valetta, Malta, 3485–3490.
Tobii Technology. 2012. Determining the Tobii I-VT Fixation Filter’s Default Values: Method Description and Results Discussion. Accessed 20 December 2013. [URL].Google Scholar logo with link to Google Scholar
Unsworth, Nash, Richard P. Heitz, Josef C. Schrock, and Randall W. Engle. 2005. “An Automated Version of the Operation Span Task.” Behavior Research Methods 37 (3): 498–505. Google Scholar logo with link to Google Scholar
Vieira, Lucas N. 2014. “Indices of Cognitive Effort in Machine Translation Post-Editing.” Machine Translation 28 (3–4): 187–216. Google Scholar logo with link to Google Scholar
. 2016. Cognitive Effort in Post-Editing of Machine Translation: Evidence from Eye Movements, Subjective Ratings, and Think-Aloud Protocols . PhD Thesis, Newcastle University.
Cited by (8)

Cited by eight other publications

Algaraady, Jeehaan & Mohammad Mahyoob
2025. Exploring ChatGPT's potential for augmenting post-editing in machine translation across multiple domains: challenges and opportunities. Frontiers in Artificial Intelligence 8 DOI logo
Carl, Michael
2025. Temporal Dynamics of Emotion and Cognition in Human Translation: Integrating the Task Segment Framework and the HOF Taxonomy. Digital Studies in Language and Literature DOI logo
Carl, Michael
2025. RETRACTED: Tracing the temporal dynamics of emotion and cognition in behavioral translation data. Translation Spaces DOI logo
Walker, Callum, Kristian Tangsgaard Hvelplund & Victoria Lei
2025. Screen eyetracking. In Research Methods in Cognitive Translation and Interpreting Studies [Research Methods in Applied Linguistics, 10], ► pp. 213 ff. DOI logo
Albl-Mikasa, Michaela, Maureen Ehrensberger-Dow, Andrea Hunziker Heeb, Caroline Lehr, Michael Boos, Matthias Kobi, Lutz Jäncke & Stefan Elmer
2020. Cognitive load in relation to non-standard language input. Translation, Cognition & Behavior 3:2 ► pp. 263 ff. DOI logo
de Faria Pires, Loïc
2020. Master’s students’ post-editing perception and strategies. FORUM. Revue internationale d’interprétation et de traduction / International Journal of Interpretation and Translation 18:1 ► pp. 26 ff. DOI logo
Hatzidaki, Anna
2019. Using experimental approaches to study translation. Translation, Cognition & Behavior 2:1 ► pp. 35 ff. DOI logo
Koponen, Maarit, Leena Salmi & Markku Nikulin
2019. A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output. Machine Translation 33:1-2 ► pp. 61 ff. DOI logo

This list is based on CrossRef data as of 15 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue