Cover not available

Article published In: Journal of Second Language Pronunciation
Vol. 10:3 (2024) ► pp.339–374

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

The impact of generative AI-powered chatbots on L2 comprehensibility

ORCiD logo with link | Iowa State University
 | Iowa State University
Published online: 25 February 2025
DOI logo with link
https://doi.org/10.1075/jslp.24053.son
Abstract
While generative AI-based chatbots expand opportunities for L2 pronunciation practice, not all are designed for language learning or provide explicit feedback. Through a comparison of two chatbots, Pronounce, which offers explicit pronunciation feedback, and Gemini, a general-purpose chatbot whose real-time transcription may serve as implicit feedback, this study explored whether practice with these chatbots had an impact on L2 English learners’ comprehensibility and whether any improvements were influenced by the presence of explicit feedback. Three groups of learners participated: two experimental groups, each practicing with one of the chatbots, and one control group. Although comprehensibility ratings indicated no statistically significant improvements at the group level based on training or the specific chatbot used, individual learners demonstrated improvements. These advancements were noted among motivated learners who completed most of their speaking sessions. Learners had positive impressions of their experience with the chatbots and believed that their practice contributed to their pronunciation improvement.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1The challenges of CAPT and ASR technology in L2 pronunciation learning
    • 2.2The emergence of AI-Based chatbots in language learning
  • 3.Methodology
    • 3.1Participants
    • 3.2Speaking practice intervention with chatbots Pronounce and Gemini
    • 3.3Data collection materials
    • 3.4Procedures
    • 3.5Data analysis
  • 4.Results
    • 4.1Impact of speaking practice with chatbots and impact of explicit feedback
    • 4.2Learners’ perceptions and beliefs about Gemini and Pronounce
      • 4.2.1Perceptions and beliefs about use of chatbots
        • 4.2.1.1Gemini
        • 4.2.1.2Pronounce
  • 5.Discussion
    • 5.1Limitations and future research
  • 6.Conclusion
  • References
References (47)
References
Alharbi, S., Alrazgan, M., Alrashed, A., Alnomasi, T., Almojel, R., & Alharbi, R. (2021). Automatic speech recognition: Systematic literature review. IEEE Access, 91, 131858–131876. Google Scholar logo with link to Google Scholar
Audacity Team. (2024). Audacity (Version 3.3.3) [Computer software]. [URL]
Bates, D., Mächler, M., Bolker, & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. Google Scholar logo with link to Google Scholar
Belda-Medina, J., & Calvo-Ferrer, J. R. (2022). Using chatbots as AI conversational partners in language learning. Applied Sciences, 12(17), 8427. Google Scholar logo with link to Google Scholar
Bibauw, S., François, T., & Desmet, P. (2019). Discussing with a computer to practice a foreign language: Research synthesis and conceptual framework of dialogue-based CALL. Computer Assisted Language Learning, 32(8), 827–877. Google Scholar logo with link to Google Scholar
Bibauw, S., Van den Noortgate, W., François, T., & Desmet, P. (2022). Dialogue systems for language learning: A meta-analysis. Language Learning & Technology, 26(1), 1–24.Google Scholar logo with link to Google Scholar
Braun, V., & Clarke, V. (2012). Thematic analysis. In H. Cooper, P. M. Camic, D. L. Long, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology, Vol. 2. Research designs: Quantitative, qualitative, neuropsychological, and biological (pp. 57–71). American Psychological Association. Google Scholar logo with link to Google Scholar
Cheng, V. C. W., Lau, V. K. T., Lam, R. W. K., Zhan, T. J., & Chan, P. K. (2020). Improving English phoneme pronunciation with automatic speech recognition using voice chatbot. In Technology in Education. Innovations for Online Teaching and Learning: 5th International Conference, ICTE 2020, Macau, China, August 19–22, 2020, Revised Selected Papers 5 (pp. 88–99). Springer Singapore.
Derwing, T. M., Rossiter, M., Munro, M. J., & Thomson, R. I. (2004). Second language fluency: Judgements on different tasks. Language Learning, 54(4), 655–679. Google Scholar logo with link to Google Scholar
Dizon, G. (2017). Using intelligent personal assistants for second language learning: A case study of Alexa. TESOL Journal, 8(4), 811–830. Google Scholar logo with link to Google Scholar
(2020). Evaluating intelligent personal assistants for L2 listening and speaking development. Language Learning & Technology, 24(1), 16–26.Google Scholar logo with link to Google Scholar
Firke, S. (2023). _janitor: Simple Tools for Examining and Cleaning Dirty Data_. R package version 2.2.0, 〈[URL]〉.
Fouz-González, J. (2020). Using apps for pronunciation training: An empirical evaluation of the English File Pronunciation app. Language Learning & Technology, 24(1), 62–85. Google Scholar logo with link to Google Scholar
Gemini. (2024). Gemini (1.5 Flash). Large Language Model. [URL]
Godwin-Jones, R. (2023). Emerging spaces for language learning: AI bots, ambient intelligence, and the metaverse. Language Learning & Technology, 27(2), 6–27.Google Scholar logo with link to Google Scholar
Golonka, E. M., Bowles, A. R., Frank, V. M., Richardson, D. L., & Freynik, S. (2014). Technologies for foreign language learning: A review of technology types and their effectiveness. Computer assisted language learning, 27(1), 70–105. Google Scholar logo with link to Google Scholar
Guskaroska, A. (2024). Exploring technology acceptance of ASR for pronunciation learning. [Unpublished doctoral dissertation]. Iowa State University. Ames, IA, USA.
Hartig, F. (2022). DHARMa: Residual diagnostics for hierarchical Multi-level / Mixed regression models. [URL]
Henrichsen, L. (2019). A System for Analyzing and Evaluating Computer-Assisted Second-Language Pronunciation-Teaching Websites and Mobile Apps. In Society for Information Technology & Teacher Education International Conference (pp. 963–968). Association for the Advancement of Computing in Education (AACE).Google Scholar logo with link to Google Scholar
Hoang, N. T., Han, D. N., & Le, D. H. (2023). Exploring Chatbot AI in improving vocational students’ English pronunciation. AsiaCALL Online Journal, 14(2), 140–155. Google Scholar logo with link to Google Scholar
Huang, W., Hew, K. F., & Fryer, L. K. (2022). Chatbots for language learning — Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237–257. Google Scholar logo with link to Google Scholar
Jeon, J. (2024). Exploring AI chatbot affordances in the EFL classroom: Young learners’ experiences and perspectives. Computer Assisted Language Learning, 37(1–2), 1–26. Google Scholar logo with link to Google Scholar
Kaiser, D. (2018). Mobile-assisted pronunciation training: The iPhone pronunciation app project. IATEFL Pronunciation Special Interest Group Journal, 581, 38–52.Google Scholar logo with link to Google Scholar
Khampusaen, D., Chanprasopchai, T., & Lao-un, J. (2023). Empowering Thai Community-based Tourism Operators: Enhancing English Pronunciation Abilities with AI-based Lessons. Journal of Mekong Societies, 19(1), 132–159.Google Scholar logo with link to Google Scholar
Kim, N. Y. (2016). Effects of Voice Chat on EFL Learners’ Speaking Ability according to Proficiency Levels. Multimedia-Assisted Language Learning, 19(4).Google Scholar logo with link to Google Scholar
Kim, H. S., Cha, Y., & Kim, N. Y. (2021). Effects of AI chatbots on EFL students’ communication skills. Korean Journal of English Language and Linguistics, 211, 712–734.Google Scholar logo with link to Google Scholar
Lenth, R. (2023). _emmeans: Estimated Marginal Means, aka Least-Squares Means_. R package version 1.8.7, 〈[URL]〉.
Levis, J. (2007). Computer technology in teaching and researching pronunciation. Annual review of applied linguistics, 271, 184–202. Google Scholar logo with link to Google Scholar
Liu, S. C., & Hung, P. Y. (2016). Teaching pronunciation with computer assisted pronunciation instruction in a technological university. Universal Journal of Educational Research, 4(9), 1939–1943. Google Scholar logo with link to Google Scholar
Martins, C. G. D. F. M., Levis, J. M., & Borges, V. M. C. (2016). The design of an instrument to evaluate software for EFL/ESL pronunciation teaching. Ilha do Desterro, 691, 141–160. Google Scholar logo with link to Google Scholar
McCrocklin, S., & Edalatishams, I. (2020). Revisiting popular speech recognition software for ESL speech. TESOL Quarterly, 54(4), 1086–1097. Google Scholar logo with link to Google Scholar
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), 100050. Google Scholar logo with link to Google Scholar
Mohammadkarimi, E. (2024). Exploring the use of artificial intelligence in promoting English language pronunciation skills. LLT Journal: A Journal on Language and Language Teaching, 27(1), 98–115. Google Scholar logo with link to Google Scholar
(2025). A guide to quantitative research methods in second language pronunciation. Routledge.Google Scholar logo with link to Google Scholar
Neri, A., Cucchiarini, C., & Strik, H. (2003). Automatic Speech Recognition for second language learning: How and why it actually works. Proceedings of the 15th International Congress of Phonetic Sciences, (pp. 1157–1160).Google Scholar logo with link to Google Scholar
Ngueajio, M. K., & Washington, G. (2022). Hey ASR system! Why aren’t you more inclusive? automatic speech recognition systems’ bias and proposed bias mitigation techniques. a literature review. In International Conference on Human-Computer Interaction (pp. 421–440). Cham: Springer Nature Switzerland.Google Scholar logo with link to Google Scholar
Pennington, M. C., & Rogerson-Revell, P. (2019). Using Technology for Pronunciation Teaching, Learning, and Assessment. In: English Pronunciation Teaching and Research. Research and Practice in Applied Linguistics. Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Pronounce (2024). Pronounce Inc. [URL]
Revelle, W. (2023). psych: Procedures for Psychological, Psychometric, and Personality Research. Northwestern University, Evanston, Illinois, USA. R package version 2.3.6, [URL]
Rogerson-Revell, P. M. (2021). Computer-assisted pronunciation training (CAPT): Current issues and future directions. RELC Journal, 52(1), 189–205. Google Scholar logo with link to Google Scholar
Tejedor-García, C., Escudero-Mancebo, D., Cámara-Arenas, E., González-Ferreras, C., & Cardeñoso-Payo, V. (2020). Assessing pronunciation improvement in students of English using a controlled computer-assisted pronunciation tool. IEEE Transactions on Learning Technologies, 13(2), 269–282. Google Scholar logo with link to Google Scholar
Thomson, R. I. (2011). Computer assisted pronunciation training: Targeting second language vowel perception improves pronunciation. CALICO Journal, 28(3), 744–765. Google Scholar logo with link to Google Scholar
Walesiak, B. (2017). Mobile pron. apps–a personal investigation. Speak Out! Journal of the IATEFL Pronunciation Special Interest Group, 571, 16–28.Google Scholar logo with link to Google Scholar
Wassink, A. B., Gansen, C., & Bartholomew, I. (2022). Uneven success: automatic speech recognition and ethnicity-related dialects. Speech Communication, 1401, 50–70. Google Scholar logo with link to Google Scholar
Weizenbaum, J. (1966). ELIZA: a computer program for the study of natural language communication between man and machine. Communications of the ACM 9(1): 36–45. Google Scholar logo with link to Google Scholar
Wickham, H., & Grolemund, G. (2017). R for data science: Visualize, model, transform, tidy, and import data. Boston, MA: O’Reilly Media. [URL]
Cited by (1)

Cited by one other publication

Jantakoon, Thada, Thiti Jantakun, Kitsadaporn Jantakun, Weerapa Pongpanich, Rungfa Pasmala, Panita Wannapiroon & Prachyanun Nilsook
2025. The effectiveness of artificial intelligence in English instruction for speaking and listening skills: A meta-analysis. Contemporary Educational Technology 17:4 ► pp. ep596 ff. DOI logo

This list is based on CrossRef data as of 21 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue