Cover not available

Article published In: Translation Spaces
Vol. 13:2 (2024) ► pp.265–292

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

The Machine Translation Post-Editing Annotation System (MTPEAS)

A standardized and user-friendly taxonomy for student post-editing quality assessment

 | UCLouvain
 | UCLouvain
ORCiD logo with link | UCLouvain
Published online: 9 September 2024
DOI logo with link
https://doi.org/10.1075/ts.24002.bod
Abstract
Machine translation post-editing quality evaluation has received relatively little attention in translation pedagogy to date. It is a time-consuming process that involves the comparison of three texts (source text, machine translation and student post-edited text) and the systematic identification and correction of students’ edits (or absence thereof) of machine translation (MT) output. There are as yet no widely available, standardized, user-friendly annotation systems for use in translator education. In this article, we address this gap by describing the Machine Translation Post-Editing Annotation System (MTPEAS). MTPEAS includes a taxonomy of seven categories that are presented in easy-to-understand terms: Value-adding edits, Successful edits, Unnecessary edits, Incomplete edits, Error-introducing edits, Unsuccessful edits, and Missing edits. We then assess the robustness of the MTPEAS taxonomy in a pilot study of 30 students’ post-edited texts and offer some preliminary findings on students’ MT error identification and correction skills.
Article outline
  • 1.Introduction: Post-editing quality evaluation in translator education
  • 2.The Machine Translation Post-Editing Annotation System (MTPEAS)
    • 2.1The MTPEAS annotation system
    • 2.2The MTPEAS taxonomy
    • 2.3The MTPEAS decision tree: A user-friendly, graphical representation of the annotation system
  • 3.Pilot Study: Data and methodology
    • 3.1Data
    • 3.2Methodology
  • 4.Pilot Study: Results and discussion
    • 4.1Inter-rater agreement on segment identification
    • 4.2Inter-rater agreement on MTPEAS annotation
    • 4.3Preliminary findings
  • 5.Concluding remarks
  • Acknowledgments
  • Notes
  • References
References (41)
References
Blain, Frédéric, Jean Senellart, Holger Schwenk, Mirko Plitt, and Johann Roturier. 2011. “Qualitative analysis of post-editing for high quality machine translation.” In Proceedings of Machine Translation Summit XIII: Papers, 164–171.Google Scholar logo with link to Google Scholar
Castagnoli, Sara, Dragos Ciobanu, Natalie Kübler, Kerstin Kunz, and Alexandra Volanschi. 2011. “Designing a learner translator corpus for training purposes.” In Corpora, language, teaching, and resources: From theory to practice, edited by Natalie Kübler, 221–248. Peter Lang.Google Scholar logo with link to Google Scholar
Castilho, Sheila, Stephen Doherty, Federico Gaspari, and Joss Moorkens. 2018. “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 9–38. Cham: Springer. Google Scholar logo with link to Google Scholar
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken. 2017. “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-editing with Students and Professional Translators.” Meta 62 (2): 245–270. Google Scholar logo with link to Google Scholar
De Almeida, Giselle. 2013. Translating the post-editor: an investigation of post-editing changes and correlations with professional experience across two Romance languages. PhD Thesis, Dublin City University.
Doherty, Stephen, and Dorothy Kenny. 2014. “The design and evaluation of a Statistical Machine Translation syllabus for translation students.” The Interpreter and Translator Trainer 8 (2): 295–315. Google Scholar logo with link to Google Scholar
Doherty, Stephen, Joss Moorkens, Federico Gaspari, and Sheila Castilho. 2018. “On education and training in translation quality assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 95–106. Cham: Springer. Google Scholar logo with link to Google Scholar
EMT Expert Group. 2022. “Competence Framework”. Accessed January 19, 2024. [URL]
Froeliger, Nicolas, Claire Larsonneur, and Giuseppe Sofo (eds). 2023. Traduction humaine et traitement automatique des langues: Vers un nouveau consensus?/Human Translation and Natural Language Processing: Forging a New Consensus? Studi e ricerche 35. Edizioni Ca’ Foscari – Venice University Press: Venice.Google Scholar logo with link to Google Scholar
Gene, Viveta, and Lucía Guerrero. 2022. “A Common Machine Translation Post-Editing Training Handbook for Academia, Clients, LSPs and Post-Editors.” GALA MTPE Training Special Interest Group.Google Scholar logo with link to Google Scholar
Ginovart Cid, Clara, and Carme Colominas Ventura. 2021. “The MT post-editing skill set: Course descriptions and educators’ thoughts.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 226–246. London/New York: Routledge.Google Scholar logo with link to Google Scholar
Granger, Sylviane, and Marie-Aude Lefer. 2023. “Learner translation corpora: Bridging the gap between learner corpus research and corpus-based translation studies.” International Journal of Learner Corpus Research 9 (1): 1–28. Google Scholar logo with link to Google Scholar
. 2021. “Translation-oriented Annotation System manual (Version 2.0).” CECL Papers 3. Louvain-la-Neuve: Centre for English Corpus Linguistics/Université catholique de Louvain. Accessed January 19, 2024. [URL]
. 2020. “The Multilingual Student Translation corpus: A resource for translation teaching and research.” Language Resources and Evaluation 541:1183–1199. Google Scholar logo with link to Google Scholar
Guerberof, Ana, and Joss Moorkens. 2019. “Machine translation and post-editing training as part of a master’s programme.” The Journal of Specialised Translation 311: 217–238.Google Scholar logo with link to Google Scholar
Hu, Ke, and Patrick Cadwell. 2016. “A Comparative Study of Post-editing Guidelines.” Baltic J. Modern Computing 4 (2): 346–353.Google Scholar logo with link to Google Scholar
Huertas Barros, Elsa, and Juliet Vine (eds). 2018. New Perspectives in Assessment in Translator Training. London: Routledge.Google Scholar logo with link to Google Scholar
Kenny, Dorothy. 2020. “Technology and translator training.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 498–515. London: Routledge.Google Scholar logo with link to Google Scholar
Kenny, Dorothy and Doherty Stephen. 2014. “Statistical machine translation in the translation curriculum: overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8 (2): 276–294. Google Scholar logo with link to Google Scholar
Konttinen, Kalle, Leena Salmi, and Maarit Koponen. 2021. “Revision and post-editing competences in translation education.” In Translation Revision and Post-editing: Industry Practices and Cognitive Processes, edited by Maarit Koponen, Brian S. Mossop, Isabelle Robert, and Giovanna Schocchera, 187–202. London/New York: Routledge.Google Scholar logo with link to Google Scholar
Koponen, Maarit. 2015. “How to teach machine translation post-editing? Experiences from a post-editing course.” In Proceedings of 4th Workshop on post-editing technology and practice, Miami, USA, October 30 to November 3, 2015, 2–15. Accessed January 19, 2024. [URL]
Koponen, Maarit and Leena Salmi. 2017. “Post-editing quality: Analysing the correctness and necessity of post-editor corrections.” Linguistica Antverpiensia, New Series: Themes in Translation Studies 161: 137–148.Google Scholar logo with link to Google Scholar
Koponen, Maarit, Leena Salmi, and Markku Nikulin. 2019. “A product and process analysis of post-editor corrections on neural, statistical and rule-based machine translation output.” Machine Translation 33 (1): 61–90. Google Scholar logo with link to Google Scholar
Kübler, Natalie, Alexandra Mestivier, and Mojca Pecman. 2022. “Using comparable corpora for translating and post-editing complex noun phrases in specialized texts. Insights from English-to-French specialized translation.” In Extending the Scope of Corpus-Based Translation Studies, edited by Sylviane Granger and Marie-Aude Lefer, 237–266. London: Bloomsbury. Google Scholar logo with link to Google Scholar
Kunilovskaya, Maria. 2015. “How far do we agree on the quality of translation?” English Studies at NBU 1 (1): 18–31. Google Scholar logo with link to Google Scholar
Landis, J. Richard, and Koch, Gary G. 1977. “The measurement of observer agreement for categorical data.” Biometrics 331: 159–174. Google Scholar logo with link to Google Scholar
Lefer, Marie-Aude, Romane Bodart, Adam Obrusník, and Justine Piette. 2023. The Post-Edit Me! project. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 493–494.Google Scholar logo with link to Google Scholar
Lefer, Marie-Aude, Romane Bodart, Justine Piette, and Adam Obrusník. 2024. “MTPE quality evaluation in translator education: the postedit.me app.” EAMT2024, Sheffield.Google Scholar logo with link to Google Scholar
Lipnevich, Anastasiya A., and Ernesto Panadero. 2021. “A review of feedback models and theories: Descriptions, definitions, and conclusions.” Frontiers in Education 61: 720195. Google Scholar logo with link to Google Scholar
Massey, Gary, Maria Piotrowska, and Mariusz Marczak. 2023. “(Re-) profiling T&I education: meeting evolution with innovation”. The Interpreter and Translator Trainer 17 (3):325–331. Google Scholar logo with link to Google Scholar
Mellinger, Christopher D. 2017. “Translators and machine translation: knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11 (4): 280–293. Google Scholar logo with link to Google Scholar
Nitzke, Jean, Anke Tardel, and Silvia Hansen-Schirra. 2019. “Training the modern translator – the acquisition of digital competencies through blended learning”. The Interpreter and Translator Trainer 13 (3): 292–306. Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2002. “Teaching post-editing: a proposal for course content.” In Proceedings of the 6th EAMT Workshop: Teaching machine translation, Manchester, England, November 14–15, 2002, 99–106. Accessed January 19, 2024. [URL]
Pavlović, Nataša, and Goranka Antunović. 2021. “Towards acquiring post-editing abilities through research-informed practical tasks.” Strani jezici: časopis za primijenjenu lingvistiku 50 (2): 185–205. Google Scholar logo with link to Google Scholar
Plaza Lara, Cristina. 2019. “SWOT Analysis of the Inclusion of Machine Translation and Post -Editing in the Master’s Degrees Offered in the EMT Network.” The Journal of Specialised Translation 311: 260–280.Google Scholar logo with link to Google Scholar
Rico, Celia, and Diana Gonzalez Pastor. 2022. “The Role of Machine Translation in Translation Education: A Thematic Analysis of Translator Educators’ Beliefs.” Translation & Interpreting 14 (1): 177–197. Google Scholar logo with link to Google Scholar
Schumacher, Perrine. 2023. La post-édition de traduction automatique en contexte d’apprentissage. PhD Thesis, University of Liège and University of Geneva.
Stasimioti, Maria, and Vilelmini Sosoni. 2021. “Investigating post-editing: A mixed-methods study with experienced and novice translators in the English-Greek language pair.” In Translation, interpreting, cognition: The way out of the box, edited by Tra&Co Group, 79–104. Berlin: Language Science Press. Google Scholar logo with link to Google Scholar
Strobl, Carola. 2022. “Revision behaviour of translation students in response to electronic feedback. Insights from a pilot study using Hypal4MUST.” Paper presented at EST conference 2022, Oslo, Norway, 22–25 June 2022.
Yamada, Masaru. 2019. “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialised Translation 31 (1): 87–106.Google Scholar logo with link to Google Scholar
Cited by (2)

Cited by two other publications

Jiang, Chunlan
2025. 2025 International Conference on Intelligent Communication Networks and Computational Techniques (ICICNCT), ► pp. 1 ff. DOI logo
Lei, Tingting & Jeroen van de Weijer
2025. Evaluating different translation methods: a case study of Chinese graduate students in an MTI program. Cogent Arts & Humanities 12:1 DOI logo

This list is based on CrossRef data as of 6 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue