Cover not available

In:Digital and Internet-Based Research Methods in Applied Linguistics
Edited by Matt Kessler
[Research Methods in Applied Linguistics 15] 2026
► pp. 362–385

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 17
Generative artificial intelligence tools

 | University of California, Merced
 | Michigan State University
Published online: 5 January 2026
DOI logo with link
https://doi.org/10.1075/rmal.15.17xu
Abstract
This chapter discusses how generative artificial intelligence (GenAI) tools, particularly large language models (LLMs) like ChatGPT, are emerging as powerful web-based tools for research purposes, such as data analysis, in applied linguistics research. While much attention has focused on pedagogical applications, we review how GenAI can be leveraged to support various stages of the research processes in empirical studies, such as instrument design, automated coding, text annotation, and qualitative data analysis. We address key concerns around validity and reliability as well as ethical considerations related to transparency, data privacy, and potential bias in AI-generated output. Given that GenAI is in the early stage of research application, we describe its current capacities and limitations based on emerging empirical research and propose promising directions for future studies.
Article outline
  • 1.Introduction
  • 2.Frequently asked research questions
    • Use of GenAI in the research workflow
    • Validity and reliability of GenAI for quantitative studies
    • Ethical concerns
  • 3.Implementation
    • Designing assessment materials and research instruments
    • Assessing speaking and writing
    • Analyzing learner data
    • Transcription and thematic data analysis
    • Corpus-based analyses and text annotation
    • Research article writing and revision
    • Validity and reliability of GenAI
  • 4.Example studies
    • Mizumoto and Eguchi (2023)
    • Pfau et al. (2023) and Xu et al. (2024)
    • Casal and Kessler (2023)
    • Kim and Lu (2024)
    • Morgan (2023)
    • Curry et al. (2024)
  • 5.Ethics and research integrity considerations
    • Privacy and confidentiality
    • Bias and transparency
    • Attribution of work
  • 6.Challenges and issues
    • Open science and transparency
    • Replication
    • Quality of the output
    • Access
  • 7.Future research directions
  • References
References (46)
References
Belal, M., She, J., & Wong, S. (2023). Leveraging ChatGPT as text annotation tool for sentiment analysis. arXiv:2306.17177. Google Scholar logo with link to Google Scholar
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610–623). ACM. Google Scholar logo with link to Google Scholar
Brand, J., Israeli, A., & Ngwe, D. (2024). Using LLMs for market research. Harvard Business School Working Paper, 23–062.Google Scholar logo with link to Google Scholar
Byrd, A., Flores, L., Green, D., Hassel, H., Johnson, S., Kirschenbaum, M., … & Mills, A. (2023). MLA-CCCC joint task force on writing and AI working paper: Overview of the issues, statement of principles, and recommendations. MLA-CCCC Joint Task Force. Retrieved on 24 September 2025 from [URL]
Cabral, S., Restrepo, D., Kanjee, Z., Wilson, P., Crowe, B., Abdulnour, R. E., Rodman, A. (2024). Clinical reasoning of a generative artificial intelligence model compared with physicians. JAMA Internal Medicine, 184(5), 581–583. Google Scholar logo with link to Google Scholar
Casal, J. E., & Kessler, M. (2023). Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing. Research Methods in Applied Linguistics, 2(3), 100068. Google Scholar logo with link to Google Scholar
Chun, J. Y., & Barley, N. (2024). A Comparative analysis of multiple-choice questions: ChatGPT-generated items vs. human-developed items. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 118–136). Iowa State University Digital Press. Google Scholar logo with link to Google Scholar
Crosthwaite, P., & Baisa, V. (2023). Generative AI and the end of corpus-assisted data-driven learning? Not so fast! Applied Corpus Linguistics, 3(3), 100066. Google Scholar logo with link to Google Scholar
Curry, N., Baker, P., & Brookes, G. (2024). Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT. Applied Corpus Linguistics, 4(1), 100082. Google Scholar logo with link to Google Scholar
Gao, S., Gales, M., & Xu, J. (2024). Detecting aberrant responses in automated L2 spoken English assessment. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 96–117). Iowa State University Digital Press. Google Scholar logo with link to Google Scholar
Iyer, A., Vojjala, S., & Andrew, J. (2024). Augmenting sentiments into Chat-GPT using facial emotion recognition. 10th International Conference on Advanced Computing and Communication Systems (ICACCS), 69–74.Google Scholar logo with link to Google Scholar
Jia, Y., & Aryadoust, V. (2024). The utility of generative Artificial Intelligence in rating interpreters’ accuracy: A case study of ChatGPT-4. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 59–72). Iowa State University Digital Press. Google Scholar logo with link to Google Scholar
Johnson, C. W. & Paulus, T. M. (2024). Generating a reflexive AI-assisted workflow for academic writing. The Qualitative Report, 29, 2772–2792. Google Scholar logo with link to Google Scholar
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. arXiv preprint arXiv:2004.09095. Google Scholar logo with link to Google Scholar
Khalifa, A., & Albadawy, M. (2024). Using artificial intelligence in academic writing and research: An essential productivity tool. Computer Methods and Programs in Biomedicine Update, 5, 100145. Google Scholar logo with link to Google Scholar
Kim, H., Baghestani, Sh., Yin, Sh., Karatay, Y., Kurt, S., Beck, J., & Karatay, L. (2024). ChatGPT for writing evaluation: Examining the accuracy and reliability of AI-generated scores compared to human raters. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 73–95). Iowa State University Digital Press. Google Scholar logo with link to Google Scholar
Kim, M., & Lu, X. (2024). Exploring the potential of using ChatGPT for rhetorical move-step analysis: The impact of prompt refinement, few-shot learning, and fine-tuning. Journal of English for Academic Purposes, 71, 101422. Google Scholar logo with link to Google Scholar
Kobak, D., Márquez, R. G., Horvát, E. Á., & Lause, J. (2024). Delving into ChatGPT usage in academic writing through excess vocabulary. Google Scholar logo with link to Google Scholar
Koltovskaia, S., Rahmati, P., & Saeli, H. (2024). Graduate students’ use of ChatGPT for academic text revision: Behavioral, cognitive, and affective engagement. Journal of Second Language Writing, 65, 101130. Google Scholar logo with link to Google Scholar
Kyle, K., Crossley, S. A., & Jarvis, S. (2021). Assessing the validity of lexical diversity using direct judgements. Language Assessment Quarterly, 18(2), 154–170. Google Scholar logo with link to Google Scholar
Lee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2023). Few-shot is enough: exploring ChatGPT prompt engineering method for automatic question generation in English education. Education and Information Technologies, 29, 11483–11515. Google Scholar logo with link to Google Scholar
Lin, Z., & Chen, H. (2024). Investigating the capability of ChatGPT for generating multiple-choice reading comprehension items. System, 123, 103344. Google Scholar logo with link to Google Scholar
Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474–496. Google Scholar logo with link to Google Scholar
Ma, Q., Crosthwaite, P., Sun, D., & Zou, D. (2024). Exploring ChatGPT literacy in language education: A global perspective and comprehensive approach. Computers and Education: Artificial Intelligence, 7, 100278. Google Scholar logo with link to Google Scholar
Marshall, D. T., & Naff, D. B. (2024). The ethics of using artificial intelligence in qualitative research. Journal of Empirical Research on Human Research Ethics, 19(3), 92–102. Google Scholar logo with link to Google Scholar
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. Google Scholar logo with link to Google Scholar
Mizumoto, A., Shintani, N., Sasaki, M., & Teng, M. F. (2024). Testing the viability of ChatGPT as a companion in L2 writing accuracy assessment. Research Methods in Applied Linguistics, 3(2), 100116.Google Scholar logo with link to Google Scholar
Mojadeddi, Z., & Rosenberg, J. (2024). Automated transcription of interviews in qualitative research using artificial intelligence: A simple guide. Journal of Surgery Research and Practice, 5, 1–6. Google Scholar logo with link to Google Scholar
Mollick, E. (2024). Co-intelligence: Living and working with AI. Portfolio/Penguin.Google Scholar logo with link to Google Scholar
Morgan, D. L. (2023). Exploring the use of artificial intelligence for qualitative data analysis: The case of ChatGPT. International Journal of Qualitative Methods, 22, 16094069231211248. Google Scholar logo with link to Google Scholar
Paulus, T. M., & Marone, V. (2024). “In minutes instead of weeks”: Discursive constructions of generative AI and qualitative data analysis. Qualitative Inquiry, 31(5). Google Scholar logo with link to Google Scholar
Pfau, A., Polio, C., & Xu, Y. (2023). Exploring the potential of ChatGPT in assessing L2 writing accuracy for research purposes. Research Methods in Applied Linguistics, 2(3), 100083. Google Scholar logo with link to Google Scholar
Polio, C., &. Yoon, H. J. (2018). The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics, 28, 165–188. Google Scholar logo with link to Google Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking in conversation. Language, 50(4), 696–735. Google Scholar logo with link to Google Scholar
Sen, M., Sen, S. N., & Sahin, T. G. (2023). A new era for data analysis in qualitative research: ChatGPT!. Shanlax International Journal of Education, 11, 1–15. Google Scholar logo with link to Google Scholar
Shin, D., & Lee, J. H. (2023). Can ChatGPT make reading comprehension testing items on par with human experts? Language Learning & Technology, 27(3), 27–40. [URL].
Smith, G., Fleisig, E., Bossi, M., Rustagi, I., & Yin, X. (2024). Standard language ideology in AI-generated language. arXiv preprint arXiv:2406.08726. Google Scholar logo with link to Google Scholar
Spring, R., & Johnson, M. (2022). The possibility of improving automated calculation of measures of lexical richness for EFL writing: A comparison of the LCA, NLTK and SpaCy tools. System, 106. Google Scholar logo with link to Google Scholar
Tankosić, A., Milak, E., Steele, C., & Dobinson, T. (2024). Meeting standards: (Re)colonial and subversive potential of AI modification. Australian Review of Applied Linguistics, 47(3), 366–382. Google Scholar logo with link to Google Scholar
Wang, P. (2019). On defining artificial intelligence. Journal of Artificial General Intelligence 10(2) 1–37. Google Scholar logo with link to Google Scholar
Wang, Z., Xie, Q., Feng, Y., Ding, Z., Yang, Z., & Xia, R. (2023). Is ChatGPT a good sentiment analyzer? A preliminary study. arXiv preprint arXiv:2304.04339. Google Scholar logo with link to Google Scholar
Xu, Y., Zhuang, J., Blair, B., Kim, A., Li, F., Thorson Hernández, R., & Plonsky, L. (2023). Modeling quality and prestige in applied linguistics journals: A bibliometric and synthetic analysis. Studies in Second Language Learning and Teaching, 13(4), 755–779. Google Scholar logo with link to Google Scholar
Xu, Y., Polio, C., & Pfau, A. (2024). Optimizing AI for assessing L2 writing accuracy: An exploration of temperatures and prompts. In C. A. Chapelle, G. H. Beckett, & J. Ranalli (Eds.), Exploring artificial intelligence in applied linguistics (pp. 151–174). Iowa State University Digital Press. Google Scholar logo with link to Google Scholar
Zhang, X., Diaz, A., Chen, Z., Wu, Q., Qian, K., Voss, E., & Yu, Z. (2024). DECOR: Improving coherence in L2 English writing with a novel benchmark for incoherence detection, reasoning, and rewriting. arXiv preprint arXiv:2406.19650. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue