Cover not available

Article published In: Translation and Interpreting Studies
Vol. 16:1 (2021) ► pp.101–123

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Measuring the usability of machine translation in the classroom context

 | Hunan University
 | Hunan University
 | Hunan University
Published online: 25 August 2020
DOI logo with link
https://doi.org/10.1075/tis.18047.yan
Abstract
Usability is a key factor for increasing adoption of machine translation. This study aims to measure the usability of machine translation in the classroom context by comparing translation students’ machine translation post-editing output with their manual translation in two comparable translation tasks. Three dimensions of usability were empirically measured: efficiency, effectiveness, and satisfaction. The findings suggest that machine translation post-editing is more efficient than human translation and post-editing produces fewer errors than human translation. While the types of errors vary, errors in terms of accuracy outnumber those related to fluency. In addition, participants perceive the amount of time and work that is saved when post-editing to be greater benefit than the overall utility of post-editing. Likewise, students report a strong desire to learn post-editing skills in training programs.
Article outline
  • Introduction
  • Relevant research
    • Usability and its assessment
    • Usability and machine translation
    • Usability assessment framework of MT
  • Methodology
    • Participants
    • Instruments
      • Translog-II
      • Questionnaire
      • Quality assessment metrics
    • Translation tasks
      • Translation direction
      • Text selection
      • Metrics for text difficulty, length, and readability
      • Scenario
      • Data collection and procedure
  • Results
    • Efficiency
    • Effectiveness
    • Error count
    • Error type
    • Satisfaction
  • Discussion
  • Conclusion
  • Acknowledgements
  • Note
  • References
References (75)
References
Bevan, Nigel, James Carter, and Susan Harker. 2015. “ISO 9241-11 revised: What have we learnt about usability since 1998?” In Human Computer Interaction: Design and Evaluation, ed. by Kurosu Masaaki, 143–151. Cham: Springer. Google Scholar logo with link to Google Scholar
Bowker, Lynne. 2020. “Fit-for-purpose translation.” In The Routledge Handbook of Translation and Technology, ed. by Minako O’Hagan, 453–468. London: Routledge.Google Scholar logo with link to Google Scholar
Bowker, Lynne and Jairo Buitrago Ciro. 2015. “Investigating the usefulness of machine translation for newcomers at the public library.” Translation and Interpreting Studies 10(2): 165–186. Google Scholar logo with link to Google Scholar
Cadwell, Patrick, Sharon O’Brien, and Carlos Teixeira. 2018. “Resistance and accommodation: Factors for the (non-) adoption of machine translation among professional translators.” Perspectives 26(3): 301–321. Google Scholar logo with link to Google Scholar
Carl, Michael. 2012. “Translog-II: a program for recording user activity data for empirical reading and writing research.” In The 8th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, et al., 21–27. Istanbul.Google Scholar logo with link to Google Scholar
Carl, Michael and Toledo Cristina Báez. 2019. “Machine translation errors and the translation process: A study across different languages.” The Journal of Specialised Translation 311: 107–132.Google Scholar logo with link to Google Scholar
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Daems, Joke, et al. 2016. “The effectiveness of consulting external resources during translation and post-editing of general text types.” In New Directions in Empirical Translation Process, ed. by Carl Michael, Bangalore Srinivas, and Schaeffer Moritz, 111–133. Cham: Springer. Google Scholar logo with link to Google Scholar
Daems, Joke, Sonia Vandepitte, Robert Hartsuiker, and Lieve Macken. 2017a. “Translation methods and experience: A comparative analysis of human translation and post-editing with student and professional translators.” Meta 62(2): 246–270. Google Scholar logo with link to Google Scholar
. 2017b. “Identifying the machine translation error types with the greatest impact on post-editing effort.” Frontiers in Psychology 81: 1282. Google Scholar logo with link to Google Scholar
Davis, Fred. 1989. “Perceived usefulness, perceived ease of use, and user acceptance of information technology.” MIS Quarterly 13(3):319–340. Google Scholar logo with link to Google Scholar
Davis, Fred, Richard Bagozzi, and Paul Warshaw. 1989. “User acceptance of computer technology: A comparison of two theoretical models.” Management Science 35(8): 982–1003. Google Scholar logo with link to Google Scholar
De Almeida, Giselle and Sharon O’Brien. 2010. “Analysing post-editing performance: Correlations with years of translation experience.” In Proceedings of the 14th Annual Conference of the European Association for Machine Translation, ed. by François Yvon and Viggo Hansen, 1–8. Saint-Raphaël.Google Scholar logo with link to Google Scholar
Doherty, Stephen and Dorothy Kenny. 2014. “The design and evaluation of a statistical machine translation syllabus for translation students.” The Interpreter and Translator Trainer 8(2): 295–315. Google Scholar logo with link to Google Scholar
Doherty, Stephen and Sharon O’Brien. 2014. “Assessing the usability of raw machine translated output: A user-centered study using eye tracking.” International Journal of Human-Computer Interaction 30(1): 40–51. Google Scholar logo with link to Google Scholar
Ducar, Cynthia and Deborah Houk Schocket. 2018. “Machine translation the L2 classroom: pedagogical solutions for making peace with Google Translate.” Foreign Language Annals 51(4): 779–795. Google Scholar logo with link to Google Scholar
Fiederer, Rebecca and Sharon O’Brien. 2009. “Quality and machine translation: A realistic objective?” The Journal of Specialised Translation 111: 52–74.Google Scholar logo with link to Google Scholar
Flanagan, Marian and Tina Paulsen Christensen. 2014. “Testing post-editing guidelines: How translation trainees interpret them and how to tailor then for translator training purposes.” The Interpreter and Translator Trainer 8(2):257–275. Google Scholar logo with link to Google Scholar
Flesch, Rudolf. 1948. “A new readability yardstick.” Journal of Applied Psychology 321: 221–223. Google Scholar logo with link to Google Scholar
. 2011. “Translating by post-editing: Is it the way forward?” Machine Translation 251: 217–237. Google Scholar logo with link to Google Scholar
Germann, Ulrich. 2008. “Yawat: Yet another world alignment tool.” In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies, 20–23. Ohio: Columbus. Google Scholar logo with link to Google Scholar
Gile, Daniel. 1994. “Methodological aspects of interpretation and translation research.” In Bridge the Gap: Empirical Research in Simultaneous Interpretation, ed. by Sylvie Lambert and Barbara Moser-Mercer, 39–56. Philadelphia, PA: John Benjamins. Google Scholar logo with link to Google Scholar
Guerberof Arenas, Ana. 2012. “Productivity and Quality in the Post-editing of Outputs from Translation memories and Machine translation.” Ph.D. dissertation. Universitat Rovira i Virgili, Tarragona.Google Scholar logo with link to Google Scholar
. 2014. “Correlations between productivity and quality when post-editing in a professional context.” Machine Translation 28 (3–4): 165–186. Google Scholar logo with link to Google Scholar
Hansen, Gyde. 2008. “The dialogue in translation process research.” In Translation and Cultural Diversity: Selected proceedings of the XVII FIT World Congress, 386–397. Shanghai: Foreign Language Press.Google Scholar logo with link to Google Scholar
. 2013. “The translation process as object of research.” In The Routledge Handbook of Translation Studies, ed. by Carmen Millán and Francesca Bartrina, 88–101. London/New York: Routledge.Google Scholar logo with link to Google Scholar
Harrati, Nouzha, et al. 2016. “Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis.” Computers in Human Behavior 611: 463–471. Google Scholar logo with link to Google Scholar
ISO9241-11. 2018. “Ergonomics of human-system interaction-Part 11: Usability: Definitions and concepts.” ISO 9241-11.Google Scholar logo with link to Google Scholar
Jia, Yanfang, Michael Carl, and Xiangling Wang. 2019. “How does the post-editing of neural machine translation compare with from-scratch translation? A product and process study.” The Journal of Specialised Translation 311: 60–85.Google Scholar logo with link to Google Scholar
Kenny, Dorothy. 2018. “Sustaining disruption? The transition from statistical to neural machine translation.” Revista Tradumàtica 161: 59–70. Google Scholar logo with link to Google Scholar
Kenny, Dorothy and Stephen Doherty. 2014. “Statistical machine translation in the translation curriculum: Overcoming obstacles and empowering translators.” The Interpreter and Translator Trainer 8(2): 276–294. Google Scholar logo with link to Google Scholar
Kingscott, Geoffrey. 2002. “Technical translation and related disciplines.” Perspectives 10(4): 247–255. Google Scholar logo with link to Google Scholar
Koponen, Maarit. 2010. “Assessing machine translation quality with error analysis.” In Electronic Proceedings of the KaTu Symposium on Translation and Interpreting Studies 41: 1–12.Google Scholar logo with link to Google Scholar
. 2015. “How to teach machine translation post-editing? Experiences from a post-editing course.” In 4th Workshop on Post-Editing Technology and Practice (WPTP4), 2–15. Miami: Florida.Google Scholar logo with link to Google Scholar
. 2016. “Is machine translation post-editing worth the effort? A survey of research into post-editing and effort.” The Journal of Specialised Translation 251: 131–147.Google Scholar logo with link to Google Scholar
Kortum, Philip and Frederick Oswald. 2017. “The impact of personality on the subjective assessment of usability.” International Journal of Human-Computer Interaction 341: 177–186. Google Scholar logo with link to Google Scholar
Krüger, Ralph. 2019. “A model for measuring the usability of computer-assisted translation tools.” In Challenging Boundaries: New Approaches to Specialized Communication, ed. by Heike Elisabeth Jüngst, Lisa Link, Klaus Schubert, and Christiane Zehrer, 93–117. Berlin: Frank & Timme.Google Scholar logo with link to Google Scholar
Lacruz, Isable, Michael Denkowski, and Alon Lavie. 2014. “Cognitive demand and cognitive effort in PE.” In Third Workshop on PE Technology and Practice, ed. by Sharon O’Brien, Michel Simard, and Lucia Specia, 73–84. AMTA.Google Scholar logo with link to Google Scholar
Lewis, James R. 2012. “Usability testing.” In Handbook of Human Factors and Ergonomics, ed. by Gavriel Salvendy, 1267–1312. New York: Wiley. Google Scholar logo with link to Google Scholar
Lexile. 2007. The Lexile Framework for Reading: Theoretical Framework and Development (Tech. Rep). Durham, NC: MetaMetrics, Inc.Google Scholar logo with link to Google Scholar
Lommel, Arle, Hans Uszkoreit, and Burchardt Aljoscha. 2014. “Multidimensional Quality Metrics (MQM): A framework for declaring and describing translation quality metrics.” Tradumàtica 121: 455–463. Google Scholar logo with link to Google Scholar
Mariana, Valerie, Troy Cox, and Alan Melby. 2015. “The multidimensional quality metrics (MQM) framework: a new framework for translation quality assessment.” The Journal of Specialised Translation 231: 137–161.Google Scholar logo with link to Google Scholar
Mellinger, Christopher D. 2017. “Translators and machine translation: Knowledge and skills gaps in translator pedagogy.” The Interpreter and Translator Trainer 11(4): 280–293. Google Scholar logo with link to Google Scholar
Mellinger, Christopher D. and Gregory M. Shreve. 2016. “Match evaluation and over-editing in a translation memory environment.” In Reembedding Translation Process Research, ed. by Ricardo Muñoz Martín, 132–148. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Mellinger, Christopher D. and Thomas A. Hanson. 2017. Quantitative Research Methods in Translation and Interpreting Studies. New York: Routledge.Google Scholar logo with link to Google Scholar
. 2018. “Interpreter traits and the relationship with technology and visibility.” Translation and Interpreting Studies 13(3): 366–392. Google Scholar logo with link to Google Scholar
MetaMetrics. 2018. About Lexile ® Measures for Reading. [URL]. Last accessed 23 May 2020.
Moorkens, Joss. 2018. “What to expect from neural machine translation: A practical in-class translation evaluation exercise.” The Interpreter and Translator Trainer 12(4): 375–387. Google Scholar logo with link to Google Scholar
Moorkens, Joss, Antonio Toral, Sheila Castilho, and Andy Way. 2018. “Translators’ perceptions of literary post-editing using statistical and neural machine translation.” Translation Space 7(2): 240–262. Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2004. “Machine translatability and post-editing effort: how do they relate?” Translating and the Computer 261:1–31.Google Scholar logo with link to Google Scholar
. 2007. “An empirical investigation of temporal and technical post-editing effort.” Translation and Interpreting Studies 2(1): 83–136. Google Scholar logo with link to Google Scholar
. 2011. “Towards predicting post-editing productivity.” Machine Translation 251: 197–215. Google Scholar logo with link to Google Scholar
Plitt, Mirko and François Masselot. 2010. “A productivity test of statistical machine translation PE in a typical localization context.” Prague Bulletin of Mathematical Linguistics 931: 7–16. Google Scholar logo with link to Google Scholar
Pym, Anthony. 2013. “Translation skill-sets in a machine-translation age.” Meta 58(3): 487–503. Google Scholar logo with link to Google Scholar
R Core Team 2018. “R: A language and environment for statistical computing.” R Foundation for Statistical Computing. Vienna. [URL]. Last accessed 23 May 2020.
Raita, Eeva and Antti Oulasvira. 2011. “Too good to be bad: Favorable product expectations boost subjective usability ratings.” Interacting with Computers 231: 363–371. Google Scholar logo with link to Google Scholar
Rossi, Caroline. 2017. “Introducing statistical machine translation in translator training: From users and perceptions to course design, and back again.” Tradumàtica 151: 48–62. Google Scholar logo with link to Google Scholar
Rossi, Caroline and Jean-Pierre Chevrot. 2019. “Uses and perceptions of machine translation at the European Commission.” The Journal of Specialised Translation 311: 201–216.Google Scholar logo with link to Google Scholar
Sakamoto, Akiko. 2019. “Unintended consequences of translation technologies: from project managers’ perspectives.” Perspectives 27(1): 58–73. Google Scholar logo with link to Google Scholar
Sánchez-Gijón, Pilar and Olga Torres-Hostench. 2014. “MT Post-editing into the mother tongue or into a foreign language? Spanish-to-English MT translation output post-edited by translation trainees.” In Proceedings of the Third Workshop on Post-editing Technology and Practice, ed. by Sharon O’Brien, Michel Simard and Lucia Specia, 5–17. Vancouver.Google Scholar logo with link to Google Scholar
Shuttleworth, Mark. 2002. “Combing MT and TM on a technology-oriented translation masters: aims and perspectives.” In Proceedings of the 6th EAMT Workshop on Teaching Machine Translation, 123–129. Manchester.Google Scholar logo with link to Google Scholar
Suojanen, Tytti, Kaisa Koskinen, and Tiina Tuominen. 2015. User-Centered Translation. London/New York: Routledge.Google Scholar logo with link to Google Scholar
Temizöz, Ö. 2016. “Postediting machine translation output: subject-matter experts versus professional translators.” Perspectives 24(4): 2–18. Google Scholar logo with link to Google Scholar
Temnikova, Irina. 2010. “Cognitive evaluation approach for a controlled language PE experiment.” In Proceedings of the 7th International Conference on Language Resources and Evaluation, ed. by Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk, Stelios Piperidis, Mike Rosner and Daniel Tapias, 3485–3490. Valletta.Google Scholar logo with link to Google Scholar
Tirkkonen-Condit, Sonja. 1990. “Professional vs. Non professional translation: A think-aloud protocol study.” In Learning, Keeping and Using Language: Selected papers from the 8th World Congress of Applied Linguistics, ed. by M. A. K. Halliday, John Gibbons, and Howard Nicholas, 381–394. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Thode, Henry. 2002. Testing for Normality. New York: Marcel Dekker. Google Scholar logo with link to Google Scholar
Trace, Jonathan, Gerriet Janssen, and Valerie Meier. 2015. “Measuring the impact of rater negotiation in writing performance assessment.” Language Testing 341: 3–22. Google Scholar logo with link to Google Scholar
Van der Heijden, Hans. 2004. “User acceptance of hedonic information systems.” MIS Quarterly 28(4): 695–704. Google Scholar logo with link to Google Scholar
Wang, Huashu. 2018. “The development of translation technology in the era of big data.” In Restructuring Translation Education: Implications from China for the Rest of the World, ed. by Feng Yue, et al., 13–26. Singapore: Springer.Google Scholar logo with link to Google Scholar
Wu, Jen-Her and Shu-Ching Wang. 2005. “What drives mobile commerce? An empirical evaluation of the revised technology acceptance model.” Information & Management 421: 719–729. Google Scholar logo with link to Google Scholar
Yamada, Masaru. 2019. “The impact of Google neural machine translation on post-editing by student translators.” The Journal of Specialized Translation 311: 87–105.Google Scholar logo with link to Google Scholar
Yang, Yanxia and Xiangling Wang. 2019. “Modeling the intention to use machine translation for student translators: An extension of technology acceptance model.” Computers & Education 1331: 116–126. Google Scholar logo with link to Google Scholar
Zaharias, Panagiotis. 2009. “Developing a usability evaluation method for e-learning applications: From functional usability to motivation to learn.” International Journal of Human-computer Interaction 25(1): 75–98. Google Scholar logo with link to Google Scholar
Zhai, Yuming, Aurélien Max, and Anne Vilnat. 2018. “Construction of a multilingual corpus annotated with translation relations.” In Proceedings of the first workshop on linguistic resources for natural language processing, ed. by Peter Machonis, Anabela Barreiro, Kristina Kocijan, and Max Silberztein, 102–111. New Mexico: Santa Fe.Google Scholar logo with link to Google Scholar
Cited by (15)

Cited by 15 other publications

Li, Xiaoye, Xiangling Wang & Wentian Lai
2025. The Usability of Neural Machine Translation in Creative-Text Post-Editing: Evidence from Users’ Performance and Perception. International Journal of Human–Computer Interaction 41:21 ► pp. 13792 ff. DOI logo
Sha, Lu, Xiaoyue Wang & Tingting Liu
2025. Understanding college students’ acceptance of machine translation in foreign language learning: an integrated model of UTAUT and task-technology fit. Humanities and Social Sciences Communications 12:1 DOI logo
Shadiev, Rustam, Ziheng Zhang & Yueh-Min Huang
2025. Effects of applying speech-enabled language translation technology to EMI lectures on students’ discussion quality and learning achievement. Educational technology research and development 73:1 ► pp. 229 ff. DOI logo
Zhang, Jia
2025. Too Tricky for Rookies? An Enquiry into Novice Translation Students’ Machine Translation Literacy. In Multilingual Education Yearbook 2025 [Multilingual Education Yearbook, ], ► pp. 85 ff. DOI logo
Abd Harbi, Nour Khaled & Rudaina Mohammed Bidaa
2024. Translating Concise Arabic Phrases for Mobile Communication. JOURNAL OF LANGUAGE STUDIES 8:11 ► pp. 245 ff. DOI logo
Chernovaty, Leonid
2024. Technohumanistic problems through the prism of online translator and interpreter training. The Journal of V.N. Karazin Kharkiv National University. Series: Foreign Philology. Methods of Foreign Language Teaching :100 ► pp. 142 ff. DOI logo
Chernovaty, Leonid Mykolayovych
2024. COVERT SIMULTANEOUS POST-EDITING IN ONLINE ASSESSMENT OF STUDENTS’ SIGHT TRANSLATION . Advanced Education 12:24 ► pp. 12 ff. DOI logo
Daems, Joke
2024. Students’ Attitudes Towards Interactive and Adaptive Translation Technology: Four years of Working with Lilt. In New Advances in Translation Technology [New Frontiers in Translation Studies, ], ► pp. 239 ff. DOI logo
Xu, Yingjie, Xiaobo Tan, Mengxuan Wang & Wenbo Zhang
2024. GoalBERT: A Lightweight Named-Entity Recognition Model Based on Multiple Fusion. Applied Sciences 14:23 ► pp. 11003 ff. DOI logo
Yang, Yanxia
2024. Understanding machine translation fit for language learning: The mediating effect of machine translation literacy. Education and Information Technologies 29:15 ► pp. 20163 ff. DOI logo
Yang, Yanxia, Runze Liu, Xingmin Qian & Jiayue Ni
2023. Performance and perception: machine translation post-editing in Chinese-English news translation by novice translators. Humanities and Social Sciences Communications 10:1 DOI logo
Gong, Yan
2022. Study on Machine Translation Teaching Model Based on Translation Parallel Corpus and Exploitation for Multimedia Asian Information Processing. ACM Transactions on Asian and Low-Resource Language Information Processing DOI logo
Zhou, Xiangyan, Xiangling Wang & Xiaodong Liu
2022. The impact of task complexity and translating self-efficacy belief on students’ translation performance: Evidence from process and product data. Frontiers in Psychology 13 DOI logo
Chernovaty, Leonid & Natalia Kovalchuk
2021. Psycholinguistic Aspects of the Development of Students’ Critical Approach to the Solution of Terminological Problems in Online Translation Learning. East European Journal of Psycholinguistics 8:2 DOI logo
[no author supplied]
2024. , DOI logo

This list is based on CrossRef data as of 6 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue