Cover not available

Article published In: FORUM
Vol. 21:2 (2023) ► pp.236–257

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

A quality assessment of Korean–English patent machine translation

Automatic and human evaluations of K2E-PAT, Patent Translate and WIPO Translate translations

 | Ewha Womans University
 | Ewha Womans University
Published online: 14 November 2023
DOI logo with link
https://doi.org/10.1075/forum.00030.lee
Abstract
This paper aims to investigate the quality Korean–English patent translations by three machine translation (MT) engines based on automatic and human evaluations of Korean to English Patent Automatic Translation (K2E-PAT), a pattern-based statistical MT; and Patent Translate and WIPO Translate, both neural MTs. For title translations, WIPO Translate scored the highest in automatic and human evaluations, while results were mixed for the other two MTs. K2E-PAT slightly outperformed Patent Translate in automatic evaluation, whereas Patent Translate outperformed K2E-PAT in human evaluation. For abstract translations, Patent Translate scored the highest in automatic evaluation, followed by WIPO Translate and K2E-PAT. In human evaluation, the ranking order was the same as that of title translations, with WIPO Translate scoring the highest on average. The results indicated correlations between automatic and human evaluations, and the NMTs subject to the current study still do not render satisfactory gist translation from Korean to English.
Résumé
Cet article se focalise sur les résultats des évaluations automatiques et humaines des traductions de brevets coréen-anglais par trois logiciels de traduction automatique : Korean to English Patent Automatic Translation (K2E-PAT), une traduction statistique basée sur des modèles ; et Patent Translate et WIPO Translate, tous deux des traductions automatiques neuronales (NMT). Les traductions des documents de brevet coréens vers l’anglais – 30 titres d’inventions et 30 abrégés – ont fait l’objet d’évaluations automatiques et humaines. L’évaluation automatique a utilisé les scores BLEU et METEOR. Dans l’évaluation humaine, quatre traducteurs de brevets professionnels ont évalué les résultats de la traduction automatique en utilisant une échelle d’évaluation de cinq points et en commentant les erreurs spécifiées. Pour les traductions de titres, WIPO Translate a obtenu les meilleures notes dans les évaluations automatiques et humaines, tandis que les résultats ont été mitigés pour les deux autres MT. K2E-PAT était légèrement supérieur à Patent Translate en évaluation automatique, tandis que Patent Translate était supérieur à K2E-PAT en évaluation humaine. Pour les traductions des abrégés, Patent Translate a obtenu les meilleures notes en évaluation automatique, suivi de WIPO Translate et de K2E-PAT. En ce qui concerne l’évaluation humaine, l’ordre de classement était le même que celui des traductions de titres, WIPO Translate ayant obtenant les meilleures notes en moyenne. Les résultats indiquent qu’il y a une forte corrélation entre les évaluations automatiques et humaines, et les NMT étudiées dans cet article n’arrivent toujours pas à traduire l’essentiel du texte coréen vers l’anglais d’une façon correcte.
Article outline
  • 1.Introduction
  • 2.Evaluation of machine translation
  • 3.Patent machine translation quality assessment
  • 4.Methods
  • 5.Results
    • 5.1Translation of titles
    • 5.2Translation of abstracts
    • 5.3Error analysis
      • 5.3.1Error types
      • 5.3.2Frequent errors in title translations
      • 5.3.3Frequent errors in abstract translations
  • 6.Discussion and conclusions
  • Notes
  • References
References (48)
References
Adams, Stephen. 2020. Information Sources in Patents. Berlin: Walter de Gruyter GmbH & Co KG. Google Scholar logo with link to Google Scholar
Banerjee, Satanjeev, and Alon Lavie. 2005. “METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.” In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, Ann Arbor, Michigan, 29 June, ed. by Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss, 65–72. New Jersey: The Association for Computational Linguistics.Google Scholar logo with link to Google Scholar
Bazrafshan, Marzieh. 2014. Semantic Features for Statistical Machine Translation. PhD diss. University of Rochester.
Brkić, Marija, Sanja Seljan, and Tomislav Vičić. 2013. “Automatic and Human Evaluation on English-Croatian Legislative Test Set.” In Computational Linguistics and Intelligent Text Processing. ed. by Alexander Gelbukh, 311–317. Berlin, Heidelberg: Springer. Google Scholar logo with link to Google Scholar
Callison-Burch, Chris, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. “(Meta-)evaluation of Machine Translation.” In Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic, 23 June, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Cameron Shaw Fordyce, 136–158. Pennsylvania: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Castilho, Sheila, et al. 2017. “Is Neural Machine Translation the New State of the Art?” The Prague Bulletin of Mathematical Linguistics 1081: 109–120. Google Scholar logo with link to Google Scholar
2018. “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari and Stephen Doherty, 9–38. Cham: Springer. Google Scholar logo with link to Google Scholar
Chatzikoumi, Eirini. 2020. “How to Evaluate Machine Translation: A Review of Automated and Human Metrics.” Natural Language Engineering 261: 137–161. Google Scholar logo with link to Google Scholar
Choi, Hyoeun, and Jieun Lee. 2017. “A Study on the Evaluation of Korean-English Patent Machine Translation–Focusing on KIPRIS K2E-PAT Translation.” Interpretation and Translation 19 (1): 139–178. Google Scholar logo with link to Google Scholar
Choi, Hyoeun, Chung-ho Lee, and Jun-ho Lee. 2023. “Evaluation of English-Korean Patent Machine Translations by a Patent-Specific NMT Engine Using AutoML.” Journal of Translation Studies 24(2): 101–130.Google Scholar logo with link to Google Scholar
Choi, YooChan. 2009. “Korean to English Patent Automatic Translation (K2E-PAT) and Cross-Lingual Retrieval on KIPRIS.” World Patent Information 311: 135–136. Google Scholar logo with link to Google Scholar
Coughlin, Deborah. 2003. “Correlating Automated and Human Assessments of Machine Translation Quality.” Paper presented at MT Summit IX, New Orleans, US.
Encyclopaedia Britannica. 2022. “Electrostriction.” Accessed April 4, 2022. [URL]
European Patent Office. 2021. “Patent Translate.” Accessed February 17, 2022. [URL]
Forcada, Mikel L. 2017. “Making Sense of Neural Machine Translation.” Translation Spaces 6 (2): 291–309. Google Scholar logo with link to Google Scholar
Kinoshita, Satoshi, Tadaaki Oshio, and Tomoharu Mitsuhashi. 2017. “Comparison of SMT and NMT trained with large Patent Corpora: Japio at WAT2017.” In Proceedings of the 4th Workshop on Asian Translation, Taipei, Taiwan, 27 November–1 December, ed. by Toshiaki Nakazawa et al., 140–145. Singapore: Asian Federation of Natural Language Processing.Google Scholar logo with link to Google Scholar
Koehn, Philipp, and Rebecca Knowles. 2017. “Six Challenges for Neural Machine Translation.” In Proceedings of the First Workshop on Neural Machine Translation, Vancouver, Canada, 4 August, ed. by Minh-Thang Luong, Alexandra Birch, Grahan Neubig, and Andrew Finch, 28–39. Pennsylvania: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Koehn, Philipp et al. 2007. “Moses: Open-Source Toolkit for Statistical Machine Translation.” Paper presented at Annual Meeting of the Association for Computational Linguistics (ACL), demonstration session.
Korea Institute of Patent Information (KIPI) (2018). “KIPRIS Brochure.” Accessed February 17, 2022. [URL]
Korea Intellectual Property Office (KIPO) and Korea Institute of Patent Information (KIPI). 2021. “Intellectual Property Statistics Service.” Accessed February 17, 2022. [URL]
Lavie, Alon, and Abhaya Agarwal. 2007. “METEOR: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments.” In Proceedings of the Second Workshop on Statistical Machine Translation, Prague, Czech Republic, 23 June, ed. by Chris Callison-Burch, Philipp Koehn, Christof Monz, and Cameron Shaw Fordyce, 228–231. Pennsylvania: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Lavie, Alon. 2013. “MT Evaluation: Human Measures and Assessment Methods.” Accessed February 24, 2022. [URL]
Lee, Jieun, and Hyoeun Choi. 2023. “A Case Study of the Evaluation of Legal and Patent Korean-English Machine Translations by a Domain-Specific NMT Engine.” Journal of Translation Studies 24(1): 9–37.Google Scholar logo with link to Google Scholar
Lommel, Arle. 2018. “Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 109–208. Cham: Springer. Google Scholar logo with link to Google Scholar
Long, Zi et al. 2016. “Translation of Patent Sentences with a Large Vocabulary of Technical Terms Using Neural Machine Translation.” In Proceedings of the 3rd Workshop on Asian Translation (WAT2016), Osaka, Japan, 11–16 December, 47–57. Pennsylvania: Association for Computational Linguistics.Google Scholar logo with link to Google Scholar
Matsutani, Yohei. 2019. “Utilization of Machine Translation in the Japan Patent Office toward Improvement of Accessibility to Patent Information.” Accessed February 17, 2022. [URL]
Nurminen, Mary. 2019. “Decision-making, risk, and gist machine translation in the work of patent professionals.” In Proceedings of the 8th Workshop on Patent and Scientific Literature Translation, Dublin, Ireland, 20 August, ed. by Takehito Utsuro, Katsuhito Sudoh, and Takashi Tsunakawa, 32–42. European Association for Machine Translation.Google Scholar logo with link to Google Scholar
Olohan, Maeve. 2015. Scientific and Technical Translation. London: Routledge. Google Scholar logo with link to Google Scholar
Panic, Milica. 2020. “Automated MT Evaluation Metrics.” Accessed February 17, 2022. [URL]
Poibeau, Thierry. 2017. Machine Translation. Cambridge: The MIT Press. Google Scholar logo with link to Google Scholar
Popović, Maja. 2017. “Comparing Language Related Issues for NMT and PBMT between German and English.” Prague Bull Math Linguist 108 (1): 209–220. Google Scholar logo with link to Google Scholar
. 2018. “Error Classification and Analysis for Machine Translation Quality Assessment.” In Translation Quality Assessment, ed. by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 129–158. Cham: Springer. Google Scholar logo with link to Google Scholar
Premoli, Valeria, Elena Murgulo, and Diego Cresceri. 2019. “MTPE in Patents: A Successful Business Story.” In Proceedings of MT Summit XVII, vol. 2, Dublin, Ireland, 19–23 August, edited by Mikel Forcada, Andy Way, Barry Haddow, and Rico Sennrich, 36–41. European Association for Machine Translation.Google Scholar logo with link to Google Scholar
Qun, Liu, and Zhang Xiaojun. 2014. “Machine Translation.” In The Routledge Encyclopedia of Translation Technology, ed. by Chan Sin-Wai, 105–119. London: Routledge.Google Scholar logo with link to Google Scholar
Rossi, Laura, and Dion Wiggins. 2013. “Applicability and Application of Machine Translation Quality Metrics in the Patent Field.” World Patent Information 351: 115–125. Google Scholar logo with link to Google Scholar
Snover, Matthew et al. 2006. “A Study of Translation Edit Rate with Targeted Human Annotation.” In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Cambridge, USA., 8–12 August. ed. by Laurie Gerber, 223–231. Association for Machine Translation in the Americas.Google Scholar logo with link to Google Scholar
Specia, Lucia, Dhwaj Raj, and Marco Turchi. 2010. “Machine Translation Evaluation versus Quality Estimation.” Machine Translation 241: 39–50. Google Scholar logo with link to Google Scholar
Tinsley, John et al. 2012. “IP Translator: Facilitating Patent Search with Machine Translation.” In Proceedings of the 10th Conference of the Association for Machine Translation in the Americas: Commercial MT User Program, San Diego, USA., 28 October–1 November. Association for Machine Translation in the Americas. Accessed July 31, 2022. [URL]
Toral, Antonio, and Víctor M. Sánchez-Cartagena. 2017. “A Multifaceted Evaluation of Neural versus Phrase-based Machine Translation for 9 Language Directions.” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, 3–7 April, edited by Mirella Lapata, Phil Blunsom, and Alexander Koller, 1063–1073. Pennsylvania: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Wang, Xiaolin, Andrew, Finch Masao Utiyama, Taro Watanabe, and Eiichiro Sumita. 2014. “The NICT translation system for IWSLT 2014.” In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, 139–142.
Wang, Dan. 2009. “Chinese to English Automatic Patent Machine Translation at SIPO.” World Patent Information 31(2): 137–139. Google Scholar logo with link to Google Scholar
World Intellectual Property Organization (WIPO). 2014. “Patent Cooperation Treaty: Yearly Review.” Accessed February 17, 2022. [URL]
World Intellectual Property Organization. 2020a. “WIPO Translate: Terms and Conditions for the Usage and User Guide.” Accessed February 17, 2022. [URL]
. 2020b. “WIPO IP Facts and Figures 2020.” Accessed February 17, 2022. [URL]
. 2020c. “World Intellectual Property Indicators 2020.” Accessed February 17, 2022. [URL]
Ying, Cheng et al. 2021. “Errors of Machine Translation of Terminology in the Patent Text from English into Chinese.” ASP Transactions on Computers 1 (1): 12–17. Google Scholar logo with link to Google Scholar
Cited by (2)

Cited by two other publications

Chen, Luyu, Lilla Varga & Milad Mehdizadkhani
2025. Taming AI for The Little Prince: a comparative analysis of NMTs and LLMs in Hungarian translation. Humanities and Social Sciences Communications DOI logo
Kim, Sunhye, Hyein Kim & Hyejin Jang
2025. A Multilingual Analysis Framework for Evaluating Technological Competitiveness in Cutting Machining Technology. Engineering Management Journal ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue