Cover not available

Article published In: Translation Spaces
Vol. 14:2 (2025) ► pp.303–330

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Missing the human touch?

A computational stylometric analysis of GPT-4 translations of online Chinese literature

ORCiD logo with link | The University of Hong Kong
ORCiD logo with link | Swinburne University of Technology
ORCiD logo with link | Swinburne University of Technology
Published online: 24 November 2025
DOI logo with link
https://doi.org/10.1075/ts.24043.yao
Abstract
Existing research suggests that machine translations of literary texts remain unsatisfactory. Such quality assessment often relies on automated metrics and subjective human ratings, with little attention to the stylistic features of machine translation. Current understanding is limited regarding the extent to which AI may transform the literary translation landscape, with implications for other critical domains for translation such as the creative industries more broadly. This pioneering study investigates the stylistic features of AI translations, specifically examining GPT-4’s performance against human translations of Chinese online literature. Our computational stylometry analysis reveals that GPT-4 translations closely mirror human translations in lexical, syntactic and content features. In addition to showing the relevance of stylometry for analysing the features of AI translation, the study provides critical insights into the implications of AI for literary translation in the posthuman tradition, where the line between machine and human translation becomes increasingly blurry.
Article outline
  • Introduction
  • Style in AI literary translation
  • Methods and data
    • Target large language model: GPT
    • Dataset: Online Chinese novels
    • Machine translation approaches
    • Stylometric analysis
  • Results and analysis
    • Lexical features
    • Syntactic features
    • Content features
  • Discussion and conclusion
  • Acknowledgement
  • Note
  • References
References (71)
References
Abbasi, Ahmed, and Hsinchun Chen. 2008. “Writeprints: A Stylometric Approach to Identity-Level Identification and Similarity Detection in Cyberspace.” ACM Transactions on Information Systems 26 (2): 1–29. Google Scholar logo with link to Google Scholar
Abu-Rayyash, Hussein. 2024. “AI Meets Comedy: Viewers’ Reactions to GPT-4 Generated Humor Translation.” Ampersand 12 (100162): 1–8. Google Scholar logo with link to Google Scholar
Burrows, John. 2002. “‘Delta’: A measure of stylistic difference and a guide to likely authorship.” Literary and Linguistic Computing 17 (3): 267–87. Google Scholar logo with link to Google Scholar
Cao, Qilin. 2021. “Accommodating Paratextual Theory to Born-Digital Literature Translation: A Case Study of ‘Wuxiaworld’.” Asia Pacific Translation and Intercultural Studies 8 (2): 148–62. Google Scholar logo with link to Google Scholar
Carl, Michael. 2021. “An Enactivist-Posthumanist Perspective on the Translation Process.” In Contesting Epistemologies in Cognitive Translation and Interpreting Studies, edited by Sandra L. Halverson and Álvaro Marín García, 176–89. Routledge. Google Scholar logo with link to Google Scholar
Castilho, Sheila, Clodagh Mallon, Rahel Meister, and Shengya Yue. 2023. “Do Online Machine Translation Systems Care for Context? What About a GPT Model?” In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 393–417. Tampere, Finland.Google Scholar logo with link to Google Scholar
Castilho, Sheila, Stephen Doherty, Federico Gaspari, and Joss Moorkens. 2018. “Approaches to Human and Machine Translation Quality Assessment.” In Translation Quality Assessment, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 19–38. Springer International Publishing. Google Scholar logo with link to Google Scholar
Chang, Jie, and Gang Zhao. 2022. “The Reader”s Visibility: Analyzing Reader’s Intervention in Fan-Based Translation on Wuxiaworld.” Translation Review 113 (1): 33–47. Google Scholar logo with link to Google Scholar
Cipriani, Anna Maria. 2023. Literary digital stylistics in translation studies. Springer. Google Scholar logo with link to Google Scholar
Cope, Bill, and Mary Kalantzis. 2024. “A Multimodal Grammar of Artificial Intelligence: Measuring the Gains and Losses in Generative AI.” Multimodality & Society 4 (2): 123–52. Google Scholar logo with link to Google Scholar
Cronin, Michael. 2020. “Translation and Posthumanism.” In The Routledge Handbook of Translation and Ethics, edited by Kaisa Koskinen and Nike K. Pokorn, 279–93. Routledge. Google Scholar logo with link to Google Scholar
Fang, Yu, and Haitao Liu. 2023. “Seeing Various Adventures through a Mirror: Detecting Translator’s Stylistic Visibility in Chinese Translations of Alice’s Adventure in Wonderland.” Digital Scholarship in the Humanities 38 (1): 50–65. Google Scholar logo with link to Google Scholar
Feng, Jin. 2013. Romancing the internet: Producing and consuming Chinese web romance. Brill. Google Scholar logo with link to Google Scholar
Fenoulhet, Jane. 2020. “The Relational Identity of the Translator.” Language and Intercultural Communication 20 (5): 497–505. Google Scholar logo with link to Google Scholar
Flynn, Peter. 2004. “Skopos Theory: An Ethnographic Enquiry.” Perspectives: Studies in Translatology 12 (4): 270–285. Google Scholar logo with link to Google Scholar
Gao, Yuan, Ruili Wang, and Feng Hou. 2023. “How to Design Translation Prompts for ChatGPT: An Empirical Study.” arXiv. Google Scholar logo with link to Google Scholar
Gómez-Adorno, Helena, Juan-Pablo Posadas-Durán, Germán Ríos-Toledo, Grigori Sidorov, and Gerardo Sierra. 2018. “Stylometry-based Approach for Detecting Writing Style Changes in Literary Texts.” Computación y Sistemas 22 (4): 1563–1572. Google Scholar logo with link to Google Scholar
Gourlay, Lesley. 2015. “Posthuman Texts: Nonhuman Actors, Mediators and the Digital University.” Social Semiotics 25 (4): 484–500. Google Scholar logo with link to Google Scholar
Guerberof-Arenas, Ana, and Antonio Toral. 2022. “Creativity in Translation: Machine Translation as a Constraint for Literary Texts.” Translation Spaces 11 (2): 184–212. Google Scholar logo with link to Google Scholar
Haverals, Wouter, Lindsey Geybels, and Vanessa Joosen. 2022. “A Style for Every Age: A Stylometric Inquiry into Crosswriters for Children, Adolescents and Adults.” Language and Literature: International Journal of Stylistics 31 (1): 62–84. Google Scholar logo with link to Google Scholar
Hayles, N. Katherine. 1999. How we became posthuman: Virtual bodies in cybernetics, literature, and informatics. University of Chicago Press. Google Scholar logo with link to Google Scholar
Hendy, Amr, Mohamed Abdelrehim, Amr Sharaf, et al. 2023. “How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation.” arXiv. [URL]
Hockx, Michel. 2015. Internet Literature in China. Columbia University Press.Google Scholar logo with link to Google Scholar
Huang, Libo. 2015. Style in translation: A corpus-based perspective. Springer. Google Scholar logo with link to Google Scholar
Ji, Meng, Pierrette Bouillon, and Mark Seligman. 2023. Translation Technology in Accessible Health Communication. Cambridge University Press. Google Scholar logo with link to Google Scholar
Jiang, Yuchen Eleanor, Tianyu Liu, Shuming Ma, Dongdong Zhang, Mrinmaya Sachan, and Ryan Cotterell. 2023. “Discourse centric evaluation of machine translation with a densely annotated parallel corpus.” arXiv. [URL]
Jiang, Yuchen, Tianyu Liu, Shuming Ma, et al. 2022. “BlonDe: An Automatic Evaluation Metric for Document-Level Machine Translation.” In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1550–65. Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Jiao, Wenxiang, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. “Is ChatGPT a Good Translator? Yes with GPT-4 as the Engine.” arXiv. Google Scholar logo with link to Google Scholar
Kang, Yong-Bin, Anthony McCosker, and Jane Farmer. 2023. “Leveraging Stylometric Analysis to Identify Unique Characteristics of Peer Support User Groups in Online Mental Health Forums.” Scientific Reports 2023 (13): 22979. Google Scholar logo with link to Google Scholar
Kenny, Dorothy. 2020. “Machine Translation.” In Routledge Encyclopedia of Translation Studies, 3rd ed., edited by Mona Baker and Gabriela Saldanha, 305–310. Routledge.Google Scholar logo with link to Google Scholar
. 2025. “Literary Machine Translation: From Taboo to Controversy.” In The Routledge Handbook of Translation Technology and Society, edited by Stefan Baumgarten and Michael Tieber, 381–395. Routledge. Google Scholar logo with link to Google Scholar
Kenny, Dorothy, and Marion Winters. 2020. “Machine Translation, Ethics and the Literary Translator’s Voice.” Translation Spaces 9 (1): 123–49. Google Scholar logo with link to Google Scholar
. 2024. “Customization, Personalization, and Style in Literary Machine Translation.” In Translation, Interpreting and Technological Change: Innovations in Research, Practice and Training, edited by Marion Winters, Sharon Deane-Cox, and Ursula Böser, 59–79. Bloomsbury. Google Scholar logo with link to Google Scholar
Kumarage, Tharindu, Joshua Garland, Amrita Bhattacharjee, Scott Ruston, and Huan Liu. 2023. “Stylometric Detection of AI-Generated Text in Twitter Timelines.” arXiv. Google Scholar logo with link to Google Scholar
Lagutina, Ksenia, Nadezhda Lagutina, Elena Boychuk, Inna Vorontsova, Elena Shliakhtina, Olga Belyaeva, Ilya Paramonov, and P. G. Demidov. 2019. “A Survey on Stylometric Text Features.” In 2019 25th Conference of Open Innovations Association (FRUCT), 184–95. IEEE. Google Scholar logo with link to Google Scholar
Lee, Changsoo. 2022. “How Do Machine Translators Measure up to Human Literary Translators in Stylometric Tests?” Digital Scholarship in the Humanities 37 (3): 813–29. Google Scholar logo with link to Google Scholar
Lee, Tong King. 2023. “Artificial Intelligence and Posthumanist Translation: ChatGPT versus the Translator.” Applied Linguistics Review. Google Scholar logo with link to Google Scholar
Li, Dang. 2021. “The Transcultural Flow and Consumption of Online Wuxia Literature through Fan-Based Translation.” Interventions 23 (7): 1041–65. Google Scholar logo with link to Google Scholar
Li, Defeng, Chunling Zhang, and Kanglong Liu. 2011. “Translation Style and Ideology: A Corpus-Assisted Analysis of Two English Translations of Hongloumeng.” Literary and Linguistic Computing 26 (2): 153–66. Google Scholar logo with link to Google Scholar
Moorkens, Joss, Antonio Toral, Sheila Castilho, and Andy Way. 2018. “Translators’ Perceptions of Literary Post-Editing Using Statistical and Neural Machine Translation.” Translation Spaces 7 (2): 240–62. Google Scholar logo with link to Google Scholar
O’Brien, Sharon. 2020. “Translation, human–computer interaction and cognition 1.” In The Routledge handbook of translation and cognition, edited by Fabio Alves and Arnt Jakobsen, 376–88. Routledge. Google Scholar logo with link to Google Scholar
. 2024. “Human-Centered Augmented Translation: Against Antagonistic Dualisms.” Perspectives 32 (3): 391–406. Google Scholar logo with link to Google Scholar
O’Thomas, Mark. 2017. “Humanum Ex Machina: Translation in the Post-Global, Posthuman World.” Target. International Journal of Translation Studies 29 (2): 284–300. Google Scholar logo with link to Google Scholar
Omar, Abdulfattah, and Yasser Gomaa. 2020. “The Machine Translation of Literature: Implications for Translation Pedagogy.” International Journal of Emerging Technologies in Learning 15 (11): 228–35. Google Scholar logo with link to Google Scholar
Ouyang, Youquan. 2023. A history of cyber literary criticism in China. Routledge.Google Scholar logo with link to Google Scholar
Peng, Keqin, Liang Ding, Qihuang Zhong, et al. 2023. “Towards Making the Most of ChatGPT for Machine Translation.” SSRN Electronic Journal. Google Scholar logo with link to Google Scholar
Ping, Yuan, and Binhua Wang. 2024. “Retranslated Chinese Classical Canon Journey to the West: A Stylometric Comparison between Julia Lovell’s Retranslation and Arthur Waley’s Translation.” Digital Scholarship in the Humanities 39 (1): 308–20. Google Scholar logo with link to Google Scholar
Ragni, Valentina, and Lucas Nunes Vieira. 2022. “What Has Changed with Neural Machine Translation? A Critical Review of Human Factors.” Perspectives 30 (1): 137–58. Google Scholar logo with link to Google Scholar
Raunak, Vikas, Arul Menezes, Matt Post, and Hany Hassan. 2023. “Do GPTs Produce Less Literal Translations?” In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 1041–50. Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Rybicki, Jan, and Magda Heydel. 2013. “The Stylistics and Stylometry of Collaborative Translation: Woolf’s Night and Day in Polish.” Literary and Linguistic Computing 28 (4): 708–717. Google Scholar logo with link to Google Scholar
Rybicki, Jan. 2006. “Burrowing into Translation: Character Idiolects in Henryk Sienkiewicz’s Trilogy and Its Two English Translations.” Literary and Linguistic Computing 21 (1): 91–103. Google Scholar logo with link to Google Scholar
. 2012. “The Great Mystery of the (Almost) Invisible Translator: Stylometry in Translation.” In Quantitative Methods in Corpus-Based Translation Studies, edited by Michael Oakes and Meng Ji, 231–248. John Benjamins. Google Scholar logo with link to Google Scholar
. 2025. “Can Machine Translation of Literary Texts Fool Stylometry?” Digital Scholarship in the Humanities 40 (1): 268–276. Google Scholar logo with link to Google Scholar
Shao, Yanjun. 2015. Wangluo wenxue de wangluoxing yu jingdianxing [The internetization and classicalization of Internet literature]. Journal of Peking University (Philosophy and Social Sciences) 52(1): 143–152.Google Scholar logo with link to Google Scholar
Siu, Sai Cheong. 2023. “ChatGPT and GPT-4 for Professional Translators: Exploring the Potential of Large Language Models in Translation.” SSRN. Google Scholar logo with link to Google Scholar
Sørbø, Marie N. 2018. Jane Austen speaks Norwegian: The challenges of literary translation. Brill. Google Scholar logo with link to Google Scholar
Sun, Yifeng, and Kanglong Liu. 2025. “Stylistic Nuances through Syntactic Complexity: A Corpus-Assisted Study of Narration and Dialogue in Two English Translations of Hongloumeng.” Applied Corpus Linguistics 5 (2): 100125. Google Scholar logo with link to Google Scholar
Thai, Katherine, Marzena Karpinska, Kalpesh Krishna, et al. 2022. “Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature.” arXiv. [URL].
Tian, Xiaoli, and Michael Adorjan. 2016. “Fandom and Coercive Empowerment: The Commissioned Production of Chinese Online Literature.” Media, Culture & Society 38 (6): 881–900. Google Scholar logo with link to Google Scholar
Toral, Antonio, and Andy Way. 2015. “Machine-Assisted Translation of Literary Text: A Case Study.” Translation Spaces 4 (2): 240–67. Google Scholar logo with link to Google Scholar
. 2018. “What Level of Quality Can Neural Machine Translation Attain on Literary Text?” In Translation Quality Assessment: From Principles to Practice, edited by Joss Moorkens, Sheila Castilho, Federico Gaspari, and Stephen Doherty, 263–87. Springer. Google Scholar logo with link to Google Scholar
Vinay, Jean-Paul, and Jean Darbelnet. 1995. Comparative Stylistics of French and English: A Methodology for Translation. Translated by Juan C. Sager and Marie-Jo Hamel. John Benjamins. Google Scholar logo with link to Google Scholar
Winters, Marion, and Dorothy Kenny. 2023. “Mark My Keywords: A Translator-Specific Exploration of Style in Literary Machine Translation.” In Computer-Assisted Literary Translation, edited by Andrew Rothwell, Andy Way, and Roy Youdale, 69–88. Routledge. Google Scholar logo with link to Google Scholar
Wu, Kan, and Dechao Li. 2022. “Are Translated Chinese Wuxia Fiction and Western Heroic Literature Similar? A Stylometric Analysis Based on Stylistic Panoramas.” Digital Scholarship in the Humanities 37 (4): 1376–93. Google Scholar logo with link to Google Scholar
Wu, Min-Hua. 2024. “Representing the Poetic Styles of Hamlet in Equivalent Chinese: Perng Ching-His’s Literary Translation of Shakespeare.” The Translator 30 (2): 265–80. Google Scholar logo with link to Google Scholar
Yamada, Masaru. 2023. “Optimizing Machine Translation through Prompt Engineering: An Investigation into ChatGPT’s Customizability.” In Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track, 195–204. Google Scholar logo with link to Google Scholar
Zaitsu, Wataru, and Mingzhe Jin. 2023. “Distinguishing ChatGPT(-3.5, -4)-Generated and Human-Written Papers through Japanese Stylometric Analysis.” PLOS ONE 18 (8): 1–12. Google Scholar logo with link to Google Scholar
Zhang, Biao, Barry Haddow, and Alexandra Birch. 2023. “Prompting Large Language Model for Machine Translation: A Case Study.” Proceedings of the 40th International Conference on Machine Learning, PMLR 2021: 41092–41110. [URL]
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue