Cover not available

Article published In: Computational Terminology
Edited by Ayla Rigouts Terryn and Patrick Drouin
[Terminology 31:1] 2025
► pp. 5–36

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

LlamATE

Automated terminology extraction using large‑scale generative language models

 | ARKHN
 | University of Tours
 | University of La Rochelle
 | Jožef Stefan International Postgraduate School | Jožef Stefan Institute
Published online: 23 May 2025
DOI logo with link
https://doi.org/10.1075/term.00082.tra
Abstract
Over the past decades, automatic term or terminology extraction (ATE), a natural language processing (NLP) task that aims to identify terms from specific domains by providing a list of candidate terms, has been challenging due to the strong influence of domain-specific differences on term definitions. Leveraging the advances of large-scale language models (LLMs), we propose LlamATE, a framework to verify the impact of domain specificity on ATE when using in-context learning prompts in open-sourced LLM-based chat models, namely Llama-2-Chat. We evaluate how well the LLM-based chat (e.g., using reinforcement learning with human feedback (RLHF)) models perform with different levels of domain-related information in the dominant language in NLP research (e.g., English) and other European languages (e.g., French, Slovene) from ACTER datasets, i.e., in-domain and cross-domain demonstrations with and without domain enunciation. Furthermore, we examine the potential of cross-lingual and cross-domain prompting to reduce the need for extensive data annotation of the target domain and language. The results demonstrate the potential of implicit in-domain learning where examples of the target domain are used as demonstrations for the prompts without specifying the domain of each example, and cross-lingual learning when knowledge is transferred from the dominant to lesser-represented European languages as for the data used to pre-train the LLMs. LlamATE also offers a valuable compromise by reducing the need for extensive data annotation, making it suitable for real-world applications where labeled corpora are scarce. The source code is publicly available at the following link: https://​github​.com​/honghanhh​/terminology2024.
Article outline
  • 1.Introduction
  • 2.Related work
    • 2.1Machine learning approaches
    • 2.2Neural approaches
  • 3.Datasets
  • 4.Methodology
    • 4.1Large language models
    • 4.2Architecture design
    • 4.3Domain transfer
    • 4.4Language transfer
    • 4.5Postprocessing steps
    • 4.6Self-verification
    • 4.7Experiment settings
    • 4.8Evaluation metrics
  • 5.Results
    • 5.1General observation
    • 5.2Verification strategies comparison
    • 5.3Monolingual vs. cross-lingual transfer comparison
    • 5.4Environmental impact
  • 6.Discussion
    • 6.1The impact of term length
    • 6.2Practical use of LLMs for lesser-represented languages
    • 6.3Limitations
  • 7.Ablations
    • 7.1Model sizes and prompt’s output designs
    • 7.2Optimal number of demonstrations
  • 8.Conclusion
  • Notes
  • References
References (57)
References
Astrakhantsev, Nikita A., Denis G. Fedorenko, and D. Yu. Turdakov. 2015. “Methods for Automatic Term Recognition in Domain-Specific Text Collections: A Survey.” Programming and Computer Software 41 (6): 336–49. Google Scholar logo with link to Google Scholar
Azé, Jérôme, Mathieu Roche, Yves Kodratoff, and Michèle Sebag. 2005. “Preference Learning in Terminology Extraction: A ROC-Based Approach.” arXiv preprint cs/0512050.Google Scholar logo with link to Google Scholar
Bay, Matthias, Daniel Bruneß, Miriam Herold, Christian Schulze, Michael Guckert, and Mirjam Minor. 2021. “Term Extraction from Medical Documents Using Word Embeddings.” In 2020 6th IEEE CiSt, 328–33. IEEE.Google Scholar logo with link to Google Scholar
Biemann, Chris, and Alexander Mehler. 2014. Text Mining: From Ontology Learning to Automated Text Processing Applications. Springer. Google Scholar logo with link to Google Scholar
Bolshakova, Elena, Natalia Loukachevitch, and Michael Nokel. 2013. “Topic Models Can Improve Domain Term Extraction.” In European Conference on Information Retrieval, 684–87. Springer. Google Scholar logo with link to Google Scholar
Cabré Castellví, M. Teresa, Rosa Estopa Bagot, and Jordi Vivaldi Palatresi. 2001. “Automatic Term Detection: A Review of Current Systems.” Recent Advances in Computational Terminology 21: 53–88. Google Scholar logo with link to Google Scholar
Conrado, Merley, Thiago Pardo, and Solange Rezende. 2013. “A Machine Learning Approach to Automatic Term Extraction Using a Rich Feature Set.” In Proceedings of the 2013 NAACL HLT Student Research Workshop, 16–22. Atlanta, Georgia, June 2013. Association for Computational Linguistics. [URL]
Conrado, Merley da Silva, Ariani Di Felippo, Thiago Alexandre Salgueiro Pardo, and Solange Oliveira Rezende. 2014. “A Survey of Automatic Term Extraction for Brazilian Portuguese.” Journal of the Brazilian Computer Society 20 (1): 1–28.Google Scholar logo with link to Google Scholar
Daille, Béatrice, Éric Gaussier, and Jean-Marc Langé. 1994. “Towards Automatic Extraction of Monolingual and Bilingual Terminology.” In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics. Google Scholar logo with link to Google Scholar
Delaunay, Julien, Hanh Thi Hong Tran, Carlos-Emiliano González-Gallardo, Georgeta Bordea, Mathilde Ducos, Nicolas Sidere, Antoine Doucet, Senja Pollak, and Olivier De Viron. 2024. “CoastTerm: A Corpus for Multidisciplinary Term Extraction in Coastal Scientific Literature.” In International Conference on Text, Speech, and Dialogue, 97–109. Springer. Google Scholar logo with link to Google Scholar
Dettmers, Tim, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. “QLoRA: Efficient Finetuning of Quantized LLMs.” Advances in Neural Information Processing Systems 361.Google Scholar logo with link to Google Scholar
Ding, Ning, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. “Few-NERD: A Few-Shot Named Entity Recognition Dataset.” arXiv preprint arXiv:2105.07464. Google Scholar logo with link to Google Scholar
El-Kishky, Ahmed, Yanglei Song, Chi Wang, Clare R. Voss, and Jiawei Han. 2014. “Scalable Topical Phrase Mining from Text Corpora.” Proceedings of the VLDB Endowment 8 (3): 305–16. Google Scholar logo with link to Google Scholar
Fedorenko, Denis, N. Astrakhantsev, and D. Turdakov. 2014. “Automatic Recognition of Domain-Specific Terms: An Experimental Evaluation.” Proceedings of the Institute for System Programming 26 (4): 55–72. Google Scholar logo with link to Google Scholar
Foo, Jody, and Magnus Merkel. 2010. “Using Machine Learning to Perform Automatic Term Recognition.” In LREC 2010 Workshop on Methods for Automatic Acquisition of Language Resources and Their Evaluation Methods, 23 May 2010, Valletta, Malta, 49–54. European Language Resources Association.Google Scholar logo with link to Google Scholar
Frantzi, Katerina T., Sophia Ananiadou, and Junichi Tsujii. 1998. “The C-Value/NC-Value Method of Automatic Recognition for Multi-Word Terms.” In International Conference on Theory and Practice of Digital Libraries, 585–604. Springer. Google Scholar logo with link to Google Scholar
Guo, Biyang, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. “How Close Is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection.”Google Scholar logo with link to Google Scholar
Han, Xiaowei, Lizhen Xu, and Feng Qiao. 2018. “CNN-BiLSTM-CRF Model for Term Extraction in Chinese Corpus.” In International Conference on Web Information Systems and Applications, 267–74. Springer. Google Scholar logo with link to Google Scholar
Hazem, Amir, Mérieme Bouhandi, Florian Boudin, and Béatrice Daille. 2020. “TermEval 2020: TALN-LS2N System for Automatic Term Extraction.” In Proceedings of the 6th International Workshop on Computational Terminology, 95–100.Google Scholar logo with link to Google Scholar
. 2022. “Cross-Lingual and Cross-Domain Transfer Learning for Automatic Term Extraction from Low-Resource Data.” In Proceedings of the Thirteenth Language Resources and Evaluation Conference, 648–662.Google Scholar logo with link to Google Scholar
ISO. 2019. Terminology Work and Terminology Science–Vocabulary. ISO 1087.Google Scholar logo with link to Google Scholar
Judea, Alex, Hinrich Schütze, and Sören Brügmann. 2014. “Unsupervised Training Set Generation for Automatic Acquisition of Technical Terminology in Patents.” In Proceedings of COLING 2014, The 25th International Conference on Computational Linguistics: Technical Papers, 290–300.Google Scholar logo with link to Google Scholar
Kageura, Kyo, and Bin Umino. 1996. “Methods of Automatic Term Recognition: A Review.” Terminology: International Journal of Theoretical and Applied Issues in Specialized Communication 3 (2): 259–89. Google Scholar logo with link to Google Scholar
Karan, Mladen, Jan Šnajder, and Bojana Dalbelo Bašić. 2012. “Evaluation of Classification Algorithms and Features for Collocation Extraction in Croatian.” In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), 657–62.Google Scholar logo with link to Google Scholar
Kocón, Jan, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocón, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski, Marcin Oleksy, Maciej Piasecki, Łukasz Radlínski, Konrad Wojtasik, Stanisław Wóźniak, and Przemysław Kazienko. 2023. “ChatGPT: Jack of All Trades, Master of None.”Google Scholar logo with link to Google Scholar
Kucza, Maren, Jan Niehues, Thomas Zenkel, Alex Waibel, and Sebastian Stüker. 2018. “Term Extraction via Neural Sequence Labeling: A Comparative Evaluation of Strategies Using Recurrent Neural Networks.” In INTERSPEECH, 2072–76. Google Scholar logo with link to Google Scholar
Lang, Christian, Lennart Wachowiak, Barbara Heinisch, and Dagmar Gromann. 2021. “Transforming Term Extraction: Transformer-Based Approaches to Multilingual Term Extraction Across Domains.” In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 3607–20. Google Scholar logo with link to Google Scholar
Le, Ngoc Tan, and Fatiha Sadat. 2021. “Multilingual Automatic Term Extraction in Low-Resource Domains.” In The International FLAIRS Conference Proceedings 341. Google Scholar logo with link to Google Scholar
Litvak, Marina, and Mark Last. 2008. “Graph-Based Keyword Extraction for Single-Document Summarization.” In Coling 2008: Proceedings of the Workshop Multi-Source Multilingual Information Extraction and Summarization, 17–24. Google Scholar logo with link to Google Scholar
Ljubešić, Nikola, Tomaž Erjavec, and Darja Fišer. 2018. “KAS-Term and KAS-Biterm: Datasets and Baselines for Monolingual and Bilingual Terminology Extraction from Academic Writing.” Digital Humanities 71.Google Scholar logo with link to Google Scholar
Maldonado, Alfredo, and David Lewis. 2016. “Self-Tuning Ongoing Terminology Extraction Retrained on Terminology Validation Decisions.” In Proceedings of The 12th International Conference on Terminology and Knowledge Engineering, 91–100.Google Scholar logo with link to Google Scholar
Nugumanova, Aliya, Darkhan Akhmed-Zaki, Madina Mansurova, Yerzhan Baiburin, and Almasbek Maulit. 2022. “NMF-Based Approach to Automatic Term Extraction.” Expert Systems with Applications 1991: 117179. Google Scholar logo with link to Google Scholar
Pavlopoulos, John, and Ion Androutsopoulos. 2014. “Aspect Term Extraction for Sentiment Analysis: New Datasets, New Evaluation Measures and an Improved Unsupervised Method.” In Proceedings of the 5th Workshop on Language Analysis for Social Media (LASM), 44–52. Google Scholar logo with link to Google Scholar
Qasemizadeh, Behrang, and Siegfried Handschuh. 2014. “Evaluation of Technology Term Recognition with Random Indexing.” In Proceedings of the Ninth International Conference on Language Resources and Evaluation.Google Scholar logo with link to Google Scholar
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. “Language Models Are Unsupervised Multitask Learners.” OpenAI Blog 1 (8): 9.Google Scholar logo with link to Google Scholar
Repar, Andraz, Vid Podpečan, Anže Vavpetič, Nada Lavrač, and Senja Pollak. 2019. “TermEnsembler: An Ensemble Learning Approach to Bilingual Term Extraction and Alignment.” Terminology. International Journal of Theoretical and Applied Issues in Specialized Communication 25 (1): 93–120.Google Scholar logo with link to Google Scholar
Rigouts Terryn, Ayla, Veronique Hoste, Patrick Drouin, and Els Lefever. 2020. “TermEval 2020: Shared Task on Automatic Term Extraction Using the Annotated Corpora for Term Extraction Research (ACTER) Dataset.” In 6th International Workshop on Computational Terminology (COMPUTERM 2020), 85–94. European Language Resources Association (ELRA).Google Scholar logo with link to Google Scholar
Rigouts Terryn, Ayla, Véronique Hoste, and Els Lefever. 2020a. “In No Uncertain Terms: A Dataset for Monolingual and Multilingual Automatic Term Extraction from Comparable Corpora.” Language Resources and Evaluation 54 (2): 385–418. Google Scholar logo with link to Google Scholar
. 2020b. “HAMLET: Hybrid Adaptable Machine Learning Approach to Extract Terminology.” Terminology, 2021.Google Scholar logo with link to Google Scholar
. 2022a. “D-terminer: Online Demo for Monolingual and Bilingual Automatic Term Extraction.” In Proceedings of the TERM21 Workshop, 33–40. Language Resources and Evaluation Conference (LREC 2022).Google Scholar logo with link to Google Scholar
. 2022b. “Tagging Terms in Text: A Supervised Sequential Labelling Approach to Automatic Term Extraction.” Terminology: International Journal of Theoretical and Applied Issues in Specialized Communication 28 (1): 157–89. Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Matej Martinc, Antoine Doucet, and Senja Pollak. 2022a. “Can Cross-Domain Term Extraction Benefit from Cross-Lingual Transfer?” In Discovery Science: 25th International Conference, DS 2022, Montpellier, France, October 10–12, 2022, Proceedings, 363–78. Springer. Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Matej Martinc, Andraz Pelicon, Antoine Doucet, and Senja Pollak. 2022b. “Ensembling Transformers for Cross-Domain Automatic Term Extraction.” In From Born-Physical to Born-Virtual: Augmenting Intelligence in Digital Libraries: 24th International Conference on Asian Digital Libraries, ICADL 2022, Hanoi, Vietnam, November 30–December 2, 2022, Proceedings, 90–100. Springer. Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Matej Martinc, Jaya Caporusso, Antoine Doucet, and Senja Pollak. 2023. “The Recent Advances in Automatic Term Extraction: A Survey.” arXiv preprint arXiv:2301.06767.Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Carlos-Emiliano Gonzalez-Gallardo, Julien Delaunay, Antoine Doucet, and Senja Pollak. 2024a. “Is Prompting What Term Extraction Needs?” In Text, Speech, and Dialogue, edited by Elmar Nöth, Aleš Horák, and Petr Sojka, 17–29. Cham: Springer Nature Switzerland. ISBN 978-3-031-70563-2. Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Matej Martinc, Andraz Repar, Nikola Ljubešić, Antoine Doucet, and Senja Pollak. 2024b. “Can Cross-Domain Term Extraction Benefit from Cross-Lingual Transfer and Nested Term Labeling?” Machine Learning, 1–30. Google Scholar logo with link to Google Scholar
Tran, Hanh Thi Hong, Matej Martinc, Antoine Doucet, and Senja Pollak. 2022c. “A Transformer-Based Sequence-Labeling Approach to the Slovenian Cross-Domain Automatic Term Extraction.” In Slovenian Conference on Language Technologies and Digital Humanities.Google Scholar logo with link to Google Scholar
Utka, Andrius. 2020. “Automatic Extraction of Lithuanian Cybersecurity Terms Using Deep Learning Approaches.” In Human Language Technologies–The Baltic Perspective: Proceedings of the Ninth International Conference Baltic HLT 2020, vol. 328, 39. IOS Press. Google Scholar logo with link to Google Scholar
Vintar, Špela. 2010. “Bilingual Term Recognition Revisited: The Bag-of-Equivalents Term Alignment Approach and Its Evaluation.” Terminology: International Journal of Theoretical and Applied Issues in Specialized Communication 16 (2): 141–58. Google Scholar logo with link to Google Scholar
Wang, Jiangyu, Chong Feng, Fang Liu, Xinyan Li, and Xiaomei Wang. 2023a. “Extract Then Adjust: A Two-Stage Approach for Automatic Term Extraction.” In CCF International Conference on Natural Language Processing and Chinese Computing, 236–47. Springer. Google Scholar logo with link to Google Scholar
Wang, Rui, Wei Liu, and Chris McDonald. 2016. “Featureless Domain-Specific Term Extraction with Minimal Labelled Data.” In Proceedings of the Australasian Language Technology Association Workshop 2016, 103–12.Google Scholar logo with link to Google Scholar
Wang, Xiao, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023. “InstructUIE: Multi-Task Instruction Tuning for Unified Information Extraction.” arXiv preprint arXiv:2304.08085.Google Scholar logo with link to Google Scholar
Wolf, Petra, Ulrike Bernardi, Christian Federmann, and Sabine Hunsicker. 2011. “From Statistical Term Extraction to Hybrid Machine Translation.” In Proceedings of the 15th Annual Conference of the European Association for Machine Translation.Google Scholar logo with link to Google Scholar
Yang, Lingpeng, Ji Donghong, Guodong Zhou, and Yu Nie. 2005. “Improving Retrieval Effectiveness by Using Key Terms in Top Retrieved Documents.” In European Conference on Information Retrieval, 169–84. Springer.Google Scholar logo with link to Google Scholar
Yuan, Yu, Jie Gao, and Yue Zhang. 2017. “Supervised Learning for Robust Term Extraction.” In 2017 International Conference on Asian Language Processing (IALP), 302–5. IEEE. Google Scholar logo with link to Google Scholar
Zhang, Ziqi, Jie Gao, and Fabio Ciravegna. 2018. “SemRE-Rank: Improving Automatic Term Extraction by Incorporating Semantic Relatedness with Personalised PageRank.” ACM Transactions on Knowledge Discovery from Data (TKDD) 12 (5): 1–41. Google Scholar logo with link to Google Scholar
Cited by (1)

Cited by one other publication

Rakotomalala, Christiane, Jean-Marie Paillat, Frédéric Feder, Angel Avadí, Laurent Thuriès, Marie-Liesse Vermeire, Jean-Michel Médoc, Tom Wassenaar, Caroline Hottelart, Lilou Kieffer, Elisa Ndjie, Mathieu Picart, Jorel Tchamgoue, Alvin Tulle, Laurine Valade, Annie Boyer, Marie-Christine Duchamp & Mathieu Roche
2025. A lexicon obtained and validated by a data-driven approach for organic residues valorization in emerging and developing countries. Frontiers in Artificial Intelligence 8 DOI logo

This list is based on CrossRef data as of 20 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue