Cover not available

Article published In: International Journal of Corpus Linguistics
Vol. 29:4 (2024) ► pp.534–561

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis

The case of apology

ORCiD logo with link | Beijing Foreign Studies University
 | Beijing Foreign Studies University
ORCiD logo with link | Sichuan International Studies University
ORCiD logo with link | University of Birmingham
Published online: 3 June 2024
DOI logo with link
https://doi.org/10.1075/ijcl.23087.yu
Abstract
Certain forms of linguistic annotation, like part of speech and semantic tagging, can be automated with high accuracy. However, manual annotation is still necessary for complex pragmatic and discursive features that lack a direct mapping to lexical forms. This manual process is time-consuming and error-prone, limiting the scalability of function-to-form approaches in corpus linguistics. To address this, our study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation. We compare GPT-3.5 (the model behind the free-to-use version of ChatGPT), GPT-4 (the model underpinning the precise mode of Bing chatbot), and a human coder in annotating apology components in English based on the local grammar framework. We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder. These results suggest that LLMs can be successfully deployed to aid pragma-discursive corpus annotation, making the process more efficient, scalable, and accessible.
Article outline
  • 1.Introduction
  • 2.Corpus annotation: Long-standing challenges, new opportunities
    • 2.1Challenges in automating pragmatic and discourse-level annotation
    • 2.2LLM-assisted corpus annotation
  • 3.Data and methods
    • 3.1The annotation task
    • 3.2Prompt design
    • 3.3Performance evaluation
  • 4.Results
    • 4.1GPT-3.5 versus GPT-4
    • 4.2GPT-4 versus a human annotator
      • 4.2.1Recognition of no apology
      • 4.2.2Recognition of apologising
      • 4.2.3Recognition of reason
      • 4.2.4Recognition of apologiser
      • 4.2.5Recognition of apologisee
      • 4.2.6Recognition of intensifier
    • 4.3Summary of findings
  • 5.Conclusion
  • Notes
  • References
References (48)
References
Baker, P., Brookes, G., & Evans, C. (2019). The language of patient feedback: A corpus linguistic study of online health communication. Routledge. Google Scholar logo with link to Google Scholar
Blum-Kulka, S., House, J., & Kasper, G. (1989). (Eds.). Cross-cultural pragmatics: Requests and apologies. Ablex Publishing Corporation.Google Scholar logo with link to Google Scholar
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … & Amodei, D. (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.). Advances in neural information processing systems 33: 34th conference on neural information processing systems (pp. 1877–1901). Neural Information Processing Systems Foundation, Inc.Google Scholar logo with link to Google Scholar
Cavasso, L., & Taboada, M. (2021). A corpus analysis of online news comments using the Appraisal framework. Journal of Corpora and Discourse Studies, (4), 1–38. Google Scholar logo with link to Google Scholar
Cheng, W., & Ching, T. (2018). ‘Not a guarantee of future performance’: The local grammar of disclaimers. Applied Linguistics, 39(3), 263–301.Google Scholar logo with link to Google Scholar
Ding, B., Qin, C., Liu, L., Chia, Y. K., Joty, S., Li, B., & Bing, L. (2023). Is GPT-3 a good data annotator? arXiv. Google Scholar logo with link to Google Scholar
Frei, J., & Kramer, F. (2023). Annotated dataset creation through large language models for non-English medical NLP. Journal of Biomedical Informatics, (145). Google Scholar logo with link to Google Scholar
Fuoli, M., & Hommerberg, C. (2015). Optimising transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora, 10(3), 315–349. Google Scholar logo with link to Google Scholar
Fuoli, M., Littlemore, J., & Turner, S. (2022). Sunken ships and screaming banshees: Metaphor and evaluation in film reviews. English Language & Linguistics, 26(1), 75–103. Google Scholar logo with link to Google Scholar
Garside, R., Leech, G., & McEnery, T. (1997). Corpus annotation: Linguistic information from computer text corpora. Routledge. Google Scholar logo with link to Google Scholar
Garside, R., & Smith, N. (1997). A hybrid grammatical tagger: CLAWS4. In R. Garside, G. Leech, & T. McEnery (Eds.), Corpus annotation: Linguistic information from computer text corpora (pp. 102–121). Routledge. Google Scholar logo with link to Google Scholar
Gilardi, F., Alizadeh, M., & Kubli, M. (2023). ChatGPT outperforms crowd-workers for text-annotation tasks. arXiv. Google Scholar logo with link to Google Scholar
He, X., Lin, Z., Gong, Y., Jin, A., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., & Chen, W. (2023). AnnoLLM: Making large language models to be better crowdsourced annotators. arXiv. Google Scholar logo with link to Google Scholar
Hunston, S. (2002). Pattern grammar, language teaching, and linguistic variation: Applications of a corpus-driven grammar. In R. Reppen, S. Fitzmaurice, & D. Biber (Eds.), Using corpora to explore linguistic variation (pp. 167–183). John Benjamins. Google Scholar logo with link to Google Scholar
(2011). Corpus approaches to evaluation: Phraseology and evaluative language. Routledge.Google Scholar logo with link to Google Scholar
Hunston, S., & Sinclair, J. (2001). A local grammar of evaluation. In S. Hunston & G. Thompson (Eds.), Evaluation in text: Authorial stance and the construction of discourse. Oxford University Press.Google Scholar logo with link to Google Scholar
Hunston, S., & Su, H. (2019). Patterns, constructions, and local grammar: A case study of ‘evaluation.’ Applied Linguistics, 40(4), 567–593. Google Scholar logo with link to Google Scholar
Kirk, J. M. (2016). The pragmatic annotation scheme of the SPICE-Ireland corpus. International Journal of Corpus Linguistics, 21(3), 299–322. Google Scholar logo with link to Google Scholar
Kolhatkar, V., Wu, H., Cavasso, L., Francis, E., Shukla, K., & Taboada, M. (2020). The SFU opinion and comments corpus: A corpus for the analysis of online news comments. Corpus Pragmatics, (4), 155–190. Google Scholar logo with link to Google Scholar
Leech, G. (1993). Corpus annotation schemes. Literary and Linguistic Computing, 8(4), 275–281. Google Scholar logo with link to Google Scholar
(1997). Introducing corpus annotation. In R. Garside, G. Leech, & T. McEnery (Eds.), Corpus annotation: Linguistic information from computer text corpora (pp. 1–18) Routledge.Google Scholar logo with link to Google Scholar
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language Processing. ACM Computing Surveys, 55(9), 1–35. Google Scholar logo with link to Google Scholar
Love, R., Dembry, C., Hardie, A., Brezina, V., & McEnery, T. (2017). The Spoken BNC2014: Designing and building a spoken corpus of everyday conversations. International Journal of Corpus Linguistics, 22(3), 319–344.Google Scholar logo with link to Google Scholar
Lutzky, U., & Kehoe, A. (2017a). “Oops, I didn’t mean to be so flippant”. A corpus pragmatic analysis of apologies in blog data. Journal of Pragmatics, (116), 27–36. Google Scholar logo with link to Google Scholar
(2017b). “I apologise for my poor blogging”: Searching for apologies in the Birmingham Blog Corpus. Corpus Pragmatics, (1), 37–56. Google Scholar logo with link to Google Scholar
Martin, J. R., & White, P. R. R. (2005). The language of evaluation: Appraisal in English. Palgrave Macmillan. Google Scholar logo with link to Google Scholar
McEnery, T., & Hardie, A. (2012). Corpus linguistics. Cambridge University Press.Google Scholar logo with link to Google Scholar
McEnery, T., & Wilson, A. (2001). Corpus linguistics: An introduction. Edinburgh University Press.Google Scholar logo with link to Google Scholar
Microsoft & OpenAI. (2023). Bing Chat (Apr-11-28-2023 version). [GPT-4 language model]. [URL]
Milà-Garcia, A. (2018). Pragmatic annotation for a multi-layered analysis of speech acts: A methodological proposal. Corpus Pragmatics, (2), 265–287. Google Scholar logo with link to Google Scholar
O’Keeffe, A. (2018). “Corpus-based function-to-form approaches”. In A. H. Jucker, K. P. Schneider & W. Bublitz (Eds.), Methods in pragmatics (pp. 587–618). Mouton de Gruyter. Google Scholar logo with link to Google Scholar
OpenAI. (2023). ChatGPT (Apr 11-28-2023 version). [Large language model]. [URL]
Page, R. (2014). Saying ‘sorry’: Corporate apologies posted on Twitter. Journal of Pragmatics, (62), 30–45. Google Scholar logo with link to Google Scholar
Põldvere, N., De Felice, R., & Paradis, C. (2022). Advice in conversation: Corpus pragmatics meets mixed methods. Cambridge University Press. Google Scholar logo with link to Google Scholar
Rayson, P., Archer, D., Piao, S., & McEnery, T. (2004). The UCREL semantic analysis system. In Proceedings of the Workshop on Beyond Named Entity Recognition: Semantic Labelling for NLP Tasks in Association with the LREC 2004 (pp. 7–12).Google Scholar logo with link to Google Scholar
Rühlemann, C., & Aijmer, K. (2014). Corpus pragmatics: Laying the foundations. In Corpus pragmatics: A handbook (pp. 1–26). Cambridge University Press. Google Scholar logo with link to Google Scholar
Simaki, V., Paradis, C., Skeppstedt, M., Sahlgren, M., Kucher, K., & Kerren, A. (2020). Annotating speaker stance in discourse: The Brexit Blog Corpus. Corpus Linguistics and Linguistic Theory, 16(2), 215–248.Google Scholar logo with link to Google Scholar
Su, H. (2017). Local grammars of speech acts: An exploratory study. Journal of Pragmatics, (111), 72–83. Google Scholar logo with link to Google Scholar
Su, H., & Zhang, L. (2020). Local grammars and discourse acts in academic writing: A case study of exemplification in Linguistics research articles. Journal of English for Academic Purposes, (43), Article 100805. Google Scholar logo with link to Google Scholar
Wei, X., Cui, X., Cheng, N., Wang, X., Zhang, X., Huang, S., Xie, P., Xu, J., Chen, Y., Zhang, M., Jiang, Y., & Han, W. (2023). Zero-shot information extraction via chatting with ChatGPT. arXiv. Google Scholar logo with link to Google Scholar
Weisser, M. (2014). Speech act annotation. In K. Aijmer & C. Rühlemann (Eds.), Corpus pragmatics: A handbook (pp. 84–110). Cambridge University Press. Google Scholar logo with link to Google Scholar
(2016). DART – The dialogue annotation and research tool. Corpus Linguistics and Linguistic Theory, 12(2), 355–388. Google Scholar logo with link to Google Scholar
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023). Harnessing the power of LLMs in practice: A survey on ChatGPT and beyond. arXiv. Google Scholar logo with link to Google Scholar
Yu, D. (2022). Cross-cultural genre analysis: Investigating Chinese, Italian and English CSR reports. Routledge.Google Scholar logo with link to Google Scholar
Zhao, T., & Kawahara, T. (2019). Joint dialog act segmentation and recognition in human conversations using attention to dialog context. Computer Speech & Language, (57), 108–127. Google Scholar logo with link to Google Scholar
Cited by (29)

Cited by 29 other publications

Diegoli, Eugenia
2026. The affective meanings of bowing in a web corpus of Japanese. Journal of Pragmatics 251 ► pp. 14 ff. DOI logo
Beck, Jacob, Lukas Malte Kemeter, Konrad Dürrbeck, Mohamed Hesham Ibrahim Abdalla & Frauke Kreuter
2025. Toward Integrating ChatGPT Into Satellite Image Annotation Workflows: A Comparison of Label Quality and Costs of Human and Automated Annotators. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 18 ► pp. 4366 ff. DOI logo
Clancy, Brian
2025.  Daniela Landert, Daria Dayter, Thomas C. Messerli and Miriam A. Locher, Corpus pragmatics (Cambridge Elements in Pragmatics). Cambridge: Cambridge University Press, 2023. Pp. 86. ISBN 9781009095082.. English Language and Linguistics ► pp. 1 ff. DOI logo
Curry, Niall, Tony McEnery & Gavin Brookes
2025. A question of alignment – AI, GenAI and applied linguistics. Annual Review of Applied Linguistics 45 ► pp. 315 ff. DOI logo
Dunivin, Zackary Okun
2025. Scaling hermeneutics: a guide to qualitative coding with LLMs for reflexive content analysis. EPJ Data Science 14:1 DOI logo
Edlinger, Moritz, Stefan E. Huber & Manuel Ninaus
2025. Analyzing player reviews with natural language processing to identify ecogames for education and research. Computers in Human Behavior Reports 20 ► pp. 100850 ff. DOI logo
Fuchs, Robert, Xinyue Yao, Peter Collins & Adam Smith
2025. Non-standard morphosyntactic variation in L2 English varieties world-wide: a corpus-based study. Lingua 322 ► pp. 103948 ff. DOI logo
Gillings, Mathew & Sylvia Jaworska
2025. How humans and machines identify discourse topics: A methodological triangulation. Applied Corpus Linguistics 5:1 ► pp. 100121 ff. DOI logo
Gillings, Mathew, Tobias Kohn & Gerlinde Mautner
2025. The rise of large language models: challenges for Critical Discourse Studies. Critical Discourse Studies 22:6 ► pp. 625 ff. DOI logo
Huang, Ding
2025. Review of Meyer (2023): English corpus linguistics: An introduction. International Journal of Corpus Linguistics 30:1 ► pp. 106 ff. DOI logo
Huang, Ding, Jiajin Xu, Yingming Song & Ruchen Yu
2025. Assessing the potential of using large language models for pragmatic annotation of historical texts. Journal of Historical Pragmatics DOI logo
Jbene, Mourad, Mourad Raif, Smail Tigani, Abdellah Chehri & Rachid Saadane
2025. Efficient Aspect-Based Sentiment Analysis for Conversational Recommendation Based on a Distilled TinyBERT Model. In Innovations in Smart Cities Applications Volume 8 [Lecture Notes in Networks and Systems, 1310], ► pp. 652 ff. DOI logo
Koželj, Martin, Iztok Podbregar, Maja Meško & Irena Nančovska Šerbec
2025. Communication with Disabled Fans at Sports Events: Approaches, Challenges, and Opportunities. Societies 15:2 ► pp. 31 ff. DOI logo
Mahmoudi-Dehaki, Mohsen & Nasim Nasr-Esfahani
2025. Automated vs. manual linguistic annotation for assessing pragmatic competence in English classes. Research Methods in Applied Linguistics 4:3 ► pp. 100253 ff. DOI logo
Morin, Cameron & Matti Marttinen Larsson
2025. Large corpora and large language models: a replicable method for automating grammatical annotation. Linguistics Vanguard DOI logo
Motger, Quim, Marc Oriol, Max Tiessler, Xavier Franch & Jordi Marco
2025. 2025 IEEE 33rd International Requirements Engineering Conference (RE), ► pp. 6 ff. DOI logo
Murai, Hajime, Ryogo Okuyama, Tomoya Kanazashi, Yuni Saito, Eiichi Sato, Masaki Tomita, Tomowa Hodosawa, Hideyuki Nishimura, Sui Sakagami, Masato Irifune, Jurin Sakamoto, Fumika Yoshii & Naoko Matsumoto
2025. Cross-genre comparison and pattern extraction for the features of utterances of story characters. Journal of the Japanese Association for Digital Humanities 8:1 ► pp. 1 ff. DOI logo
Nguyen, Dong
2025. Collaborative Growth: When Large Language Models Meet Sociolinguistics. Language and Linguistics Compass 19:2 DOI logo
Qamar, Md. Tauseef, Shahab Saquib Sohail, Gunjan Ansari & Chandni Saxena
2025. The Language of Nuance: Exploring the Limits of Large Language Models in Handling Ambiguity. In Big Data Analytics in Astronomy, Science, and Engineering [Lecture Notes in Computer Science, 15546], ► pp. 180 ff. DOI logo
Schofield, Alexandra, Siqi Wu, Theo Bayard de Volo, Tatsuki Kuze, Alfredo Gomez & Sharifa Sultana
2025. "My Very Subjective Human Interpretation": Domain Expert Perspectives on Navigating the Text Analysis Loop for Topic Models. Proceedings of the ACM on Human-Computer Interaction 9:1 ► pp. 1 ff. DOI logo
Shin, Eunhye
2025. Co‐Coding Classroom Dialogue: A Single Researcher Case Study of ChatGPT‐Assisted Analysis in Science Education. Journal of Computer Assisted Learning 41:4 DOI logo
Su, Hang & Jun Ye
2025. Large Language Models for Automating Fine-grained Speech Act Annotation: A Critical Evaluation of GPT-4o and DeepSeek. Corpus Pragmatics 9:4 ► pp. 463 ff. DOI logo
Xu, Jinfen & Fan Ye
2025. Two decades of research on ELT textbook content: A bibliometric and content analysis. Studies in Second Language Learning and Teaching DOI logo
Yu, Danni
2025. Towards LLM-assisted move annotation: Leveraging ChatGPT-4 to analyse the genre structure of CEO statements in corporate social responsibility reports. English for Specific Purposes 78 ► pp. 33 ff. DOI logo
Hara, Kotaro, Rosiana Natalie, Wei Soon Cheong, Jingjing Gu & Qianli Xu
2024. The 26th International ACM SIGACCESS Conference on Computers and Accessibility, ► pp. 1 ff. DOI logo
İşbilen, Nihal
2024. Özür Dileme Stratejileri ve Dindarlık Üzerine Ampirik Bir Araştırma. Marmara Üniversitesi İlahiyat Fakültesi Dergisi 67:67 ► pp. 337 ff. DOI logo
Yu, Danni, Marina Bondi & Ken Hyland
2024. Can GPT-4 learn to analyse moves in research article abstracts?. Applied Linguistics DOI logo
Yu, Danni, Hang Su & Marina Bondi
2024. Developing local grammars of speech acts in Italian: The case of apology. Lingua 299 ► pp. 103672 ff. DOI logo
[no author supplied]
2025. Local Grammar Approaches to Speech Act Studies [Studies in Corpus Linguistics, 123], DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue