Article published In: Journal of Second Language Pronunciation
Vol. 4:1 (2018) ► pp.129–153
Computer-assisted visual articulation feedback in L2 pronunciation instruction
A review
Published online: 31 May 2018
https://doi.org/10.1075/jslp.00006.bli
https://doi.org/10.1075/jslp.00006.bli
Abstract
Language learning is a multimodal endeavor; to improve their pronunciation in a new language, learners access not only auditory information about speech sounds and patterns, but also visual information about articulatory movements and processes. With the development of new technologies in computer-assisted pronunciation training (CAPT) comes new possibilities for delivering feedback in both auditory and visual modalities. The present paper surveys the literature on computer-assisted visual articulation feedback, including direct feedback that provides visual models of articulation and indirect feedback that uses visualized acoustic information as a means to inform articulation instruction. Our focus is explicitly on segmental features rather than suprasegmental ones, with visual feedback conceived of as providing visualizations of articulatory configurations, movements, and processes. In addition to discussing types of visual articulation feedback, we also consider the criteria for effective delivery of feedback, and methods of evaluation.
Keywords: multimodality, visual feedback, articulation, CAPT, segmental features
Article outline
- 1.Introduction
- 2.Situating visual articulation feedback in the context of L2 learning
- 3.Types of computer-assisted visual articulation feedback
- 3.1Indirect feedback using visualized acoustic information
- 3.2Direct feedback using visualized articulatory information
- 3.2.1Ultrasound-based feedback
- 3.2.2Feedback using intra-oral techniques
- 3.3Simulation approaches
- 4.Criteria for the effective use of visual articulation feedback
- 5.Methods of evaluation and reporting
- 6.Summary and conclusions
- Acknowledgements
References
References (67)
Abberton, E., & Fourcin, A. J. (1975). Visual feedback and the acquisition of intonation. In E. H. Lenneberg & E. Lenneberg (Eds.). Foundations of language development: A multidisciplinary approach (Vol. 21, pp. 157–165). Paris: UNESCO.
Akahane-Yamada, R., McDermott, E., Adaichi, T., Kawahara, H., & Pruitt, J. S. (1998). Computer-based second language production training by using spectrographic representation and HMM-based speech recognition scores. Paper presented at the 1998 International Conference on Spoken Language Processing, Sydney, Australia. Retrieved from <[URL]> (16 December, 2015).
Aliaga-García, C., & Mora, J. C. (2009). Assessing the effects of phonetic training on L2 sound perception and production. In M. A. Watkins, A. S. Rauber, & B. O. Baptista (Eds.), Recent research in second language phonetics/phonology: Perception and production (pp. 2–31). Newcastle upon Tyne: Cambridge Scholars.
Anderson, F. (1960). An experimental pitch indicator for training deaf scholars. Journal of the Acoustical Society of America, 32(8), 1065–1074.
Badin, P., Ben Youssef, A., Bailly, G., Elisei, F., & Hueber, T. (2010). Visual articulatory feedback for phonetic correction in second language learning. Proceedings of the Workshop on Second Language Studies: Acquisition, Learning, Education, and Technology, 1–10.
Ballard, K. J., Smith, H. D., Paramatmuni, D., McCabe, P., Theodoros, D. G., & Murdoch, B. E. (2012). Amount of kinematic feedback affects learning of speech motor skills. Motor Control, 161, 106–119.
Bernhardt, B., Gick, B., Bacsfalvi, P., & Ashdown, J. (2003). Speech habilitation of hard of hearing adolescents using electropalatography and ultrasound as evaluated by trained listeners. Clinical Linguistics & Phonetics, 17(3), 199–216.
Bernhardt, B., Gick, B., Bacsfalvi, P., & Adler-Bock, M. (2005). Ultrasound in speech therapy with adolescents and adults. Clinical Linguistics & Phonetics, 19(6/7), 605–617.
Boersma, P. (2001). Praat. A system for doing phonetics by computer. Glot International, 5(9/10), 341–345.
Bruce, C. J., Spittell, P. C., Montgomery, S. C., Bailey, K. R., Tajik, A. J., & Seward, J. B. (2000). Ultrasound imager: Abdominal aortic aneurysm screening. Journal of the American Society of Echocardiography, 131, 674–679.
Carey, M. (2004). CALL visual feedback for pronunciation of vowels: Kay Sona-Match. CALICO Journal, 21(3), 571–601.
Catford, J. C. & Pisoni, D. B. (1970). Auditory versus articulatory training in exotic sounds. The Modern Language Journal, 54(7), 477–481.
(1998). Signal analysis software for teaching discourse intonation. Language Learning & Technology, 2(1), 61–77.
(2002). Discourse intonation in L2: From theory and research to practice (Language Learning and Teaching 1). Amsterdam: John Benjamins.
(2013). Computer-assisted pronunciation teaching. In C. Chapelle (ed). The encyclopedia of applied linguistics. Oxford: Blackwell.
Clarius. (2016). Wireless, handheld ultrasound for iOS and Android debuts. [Press release]. Retrieved from <[URL]>.
Cleland, J., Scobbie, J. M., Nakai, S., & Wrench, A. (2015). Helping children learn non-native articulations: the implications for ultrasound-based clinical intervention. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from <[URL]> (12 August, 2015).
de Bot, C. L. J. (1980). The role of feedback and feedforward in the teaching of pronunciation. System, 81, 35–45.
Demenko, G., Wagner, A., & Cylwik, N. (2010). The use of speech technology in foreign language pronunciation training. Archives of Acoustics, 35(3), 309–329.
Dowd, A., Smith, J., & Wolfe, J. (1997). Learning to pronounce vowel sounds in a foreign language using acoustic measurements of the vocal tract as feedback in real time. Language and Speech, 41(1), 1–20.
Engwall, O. (2012). Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher. Computer Assisted Language Learning, 25(1), 37–64.
Gick, B., Bernhardt, B., Bacsfalvi, P., & Wilson, I. (2008). Ultrasound imaging applications in second language acquisition. In J. G. Hansen Edwards & M. L. Zampini (Eds.), Phonology and second language acquisition (pp. 309–322). Amsterdam: John Benjamins.
Hardison, D. M. (2004). Generalization of computer-assisted prosody training: Quantitative and qualitative findings. Language Learning & Technology, 81, 34–52.
Hincks, R. (2015). Technology and learning pronunciation. In M. Reed & J. M. Levis (Eds.), The handbook of English pronunciation (pp. 505–519). Hoboken, NJ: Wiley and Sons.
Jenson, P. G., & Westermeier, F. X. (1968). The effect of visual feedback on pronunciation in foreign language learning. Retrieved from <[URL]> (29 August, 2015).
Kalikow, D. N., & Swets, J. A. (1972). Experiments with computer-controlled displays in second-language learning. IEEE Transactions on Audio and Electroacoustics, AU-20(1), 23–28.
Kartushina, N., Hervais-Adelman, A., Frauenfelder, U. H., & Golestani, N. (2015). The effect of phonetic production training with visual feedback on the perception and production of foreign speech sounds. Journal of the Acoustical Society of America, 138(2), 817–832.
Katz, W., Campbell, T., Wang, J., Farrar, E., Eubanks, J. C., Balasubramanian, A., Prabhakaran, B., & Rennaker, R. (2014). Opti-Speech: A real-time, 3D visual feedback system for speech training. In Proceedings of Interspeech 2014, Singapore (pp.1174–1178). Retrieved from <[URL]> (22 January, 2016).
Katz, W. F., & Mehta, S. (2015). Visual feedback of tongue movement for novel speech sound learning. Frontiers in Human Neuroscience, 91, 612. .
Kelsey, C. A., Minifie, F. D., & Hixon, T. J. (1969). Applications of ultrasound in speech research. Journal of Speech, Language, and Hearing Research, 12(3), 564–575
Lambacher, S. (1999). A CALL tool for improving second language acquisition of English consonants by Japanese learners. Computer Assisted Language Learning, 12(2), 137–156.
Lee, J., Jang, J., & Plonksy, L. (2015). The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics, 36(3), 345–355.
Léon, P. R., & Martin, P. (1972). Applied linguistics and the teaching of intonation. The Modern Language Journal, 56(3), 139–144.
Levis, J. M. & Pickering, L. (2004). Teaching intonation in discourse using speech visualization technology. System, 32(4), 505–524.
Levitt, J. S., & Katz, W. F. (2007). Augmented visual feedback in second language learning: training Japanese post-alveolar flaps to American English speakers. Journal of the Acoustical Society of America, 122(5), 2996.
Massaro, D. W., & Light, J. (2003). Read my tongue movements: bimodal learning to perceive and produce non-native speech /r/ and /l/. Proceedings of the 8th European Conference on Speech Communication and Technology.
Mattheyses, W. & Verhelst, W. (2015). Audiovisual speech synthesis: An overview of the state- of-the-art. Speech Communication, 661, 182–217.
Moisik, S. R., Esling, J. H., Bird, S., & Lin, H. (2011). Evaluating laryngeal ultrasound to study larynx state and height. In W. S. Lee & E. Zee (Eds.), Proceedings of the 17th International Congress of Phonetic Sciences Hong Kong (pp. 136–139).
Molholt, G. (1988). Computer-assisted instruction in pronunciation for Chinese speakers of American English. TESOL Quarterly, 22(1), 91–111.
(1990). Spectrographic analysis and patterns in pronunciation. Computers and the Humanities, 24(1/2), 81–92.
Navarra, J., & Soto-Faraco, S. (2007). Hearing lips in a second language: Visual articulatory information enables the perception of second language sounds. Psychological Research 711, 4–12.
Neri, A., Cucchiarini, C., Strik, H., & Boves, L. (2002). The pedagogy-technology interface in computer-assisted pronunciation training. Computer-Assisted Language Learning, 21(5), 393–408.
Noguchi, M., Yamane, N., Tsuda, A., Kazama, M., Kim, B., & Gick, B. (2015). Towards protocols for L2 pronunciation training using ultrasound imaging. Poster presentation at the 7th annual Pronunciation in Second Language Learning and Teaching (PSLLT) Conference. Dallas, TX, October 2015.
Olson, D. J. (2014a). Phonetics and technology in the classroom: A practical approach to using speech analysis software in second-language pronunciation instruction. Hispania, 97(1), 47–68.
(2014b). Benefits of visual feedback on segmental production in the L2 classroom. Language Learning and Technology, 18(3), 173–192.
Öster, A. -M. (1997). Auditory and visual feedback in spoken L2 teaching. Reports from the Department of Phonetics, Umeå University (PHONUM), 41, 145–148.
Ouni, S. (2014). Tongue control and its implication in pronunciation training. Computer Assisted Language Learning, 27(5), 439–453.
Patten, I., & Edmonds, L. A. (2015). Effect of training Japanese L1 speakers in the production of American English /r/ using spectrographic visual feedback. Computer Assisted Language Learning, 28(3), 241–259.
Pillot-Loiseau, C., Kamiyama, T., & Kocjančič Antolík, T. (2015). French /y/-/u/ contrast in Japanese learners with/without ultrasound feedback: vowels, non-words and words. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved on from <[URL]> (12 August, 2015).
Quintana-Lara, M. (2014). Effect of acoustic spectrographic instruction on production of English /i/ and /ɪ/ by Spanish pre-service English teachers. Computer Assisted Language Learning, 27(3), 207–227.
R Core Team (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
Saito, K. (2007). The influence of explicit pronunciation instruction on pronunciation in EFL settings: the case of English vowels and Japanese learners of English. The Linguistics Journal, 3(3), 16–40.
Schwartz, B. (1993). On explicit and negative data effecting and affecting competence and linguistic behavior. Studies in Second Language Acquisition, 151, 147–163.
Stone, M. (2005). Preface to the special issue on ultrasound imaging of the tongue. Clinical Linguistics & Phonetics, 19(6–7), 453–454.
Suemitsu, A., Dang, J., Ito, T., & Tiede, M. (2015). A real-time articulatory visual feedback approach with target presentation for second language pronunciation learning. Journal of the Acoustical Society of America, 138(4), EL382–EL387.
Tateishi, M., & Winters, S. (2013). Does ultrasound training lead to improved perception of a non-native sound contrast? Evidence from Japanese learners of English. Paper presented at the 2013 meeting of the Canadian Linguistic Association, Victoria, BC, Canada. Retrieved from <[URL]> (12 August, 2015).
Thomson, R., & Derwing, T. (2014). The effectiveness of L2 pronunciation instruction: A narrative review. Applied Linguistics, 36(3): 326–344.
Tilsen, S., Das, D., & McKee, B. (2015). Real-time articulatory biofeedback with electromagnetic articulography. Linguistics Vanguard, 1(1), 39–55. .
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. System, 161: 255–272.
Tsui, H. M. (2012). Ultrasound speech training for Japanese adults learning English as a second language (Unpublished MSc thesis). University of British Columbia.
Vardanian, R. M. (1964). Teaching English intonation through oscilloscope displays. Language Learning, 14(3–4), 109–117.
Wilson, I. (2014). Using ultrasound for teaching and researching articulation. Acoustical Science and Technology, 35(6), 285–289.
Wilson, I., & Gick, B. (2006). Ultrasound technology and second language acquisition research. In M. Grantham O’Brien, C. Shea, & J. Archibald (Eds.), Proceedings of the 8th Generative Approaches to Second Language Acquisition Conference (GASLA 2006) (pp. 148–152). Somerville, MA: Cascadilla Proceedings Project.
Wojtczak, J., & Bonadonna, P. (2013). Pocket mobile smartphone system for the point-of-care submandibular ultrasonography. The American Journal of Emergency Medicine, 311, 573–577.
Wu, Y., Gendrot, C., Hallé, P., & Adda-Decker, M. (2015). On improving the pronunciation of French /r/ in Chinese learners by using real-time ultrasound visualization. Paper presented at the 2015 International Conference of Phonetic Sciences, Glasgow, Scotland. Retrieved from <[URL]> (12 August, 2015).
Cited by (39)
Cited by 39 other publications
Aalto, Eija M.A., Hana Ben Asker, Lucie Ménard, Walcir Cardoso & Catherine Laporte
Chen, Hsueh Chu, Qianwen Han & Jing Xuan Tian
Gómez González, María de los Ángeles & Rebeca García Muras
Hirschi, Kevin, Okim Kang, Mu Yang, John H. L. Hansen & Kyle Beloin
John, Paul, Carol Johnson & Walcir Cardoso
2025. Exploring automatic speech recognition for corrective and confirmative pronunciation feedback. Journal of Second Language Pronunciation 11:2 ► pp. 213 ff.
Kolesnichenko, Marina & Vitalii Kapitan
Phuong, Thi Duyen & Thi Thanh Huyen Phuong
Toyama, Michiko & Tomoko Hori
Wen, Yun & Xinyu Guo
Bahi, Halima, Bilal Dendani & Meriem Lounis
Bird, Sonya, Bryan Gick & Maida Percival
Casserly, Elizabeth D. & Francesca R. Marino
Kocjančič, Tanja, Tomáš Bořil & Susanna Hofmann
Ma, Qing, Fang Mei & Bojie Qian
Shahid, Muhammad Suhaib, Andrew P French, Michel F Valstar & Gleb E Yakubov
Sun, Yan
CENGİZ, Behice Ceyda
Schuhmann, Katharina S., Sarah Schaech & Cain Catto
Chun, Dorothy M. & Yan Jiang
Demir, Yusuf & Galip Kartal
Kühnert, Barbara & Claire Pillot-Loiseau
Angelovska, Tanja
2021. When a psycholinguist enters the multilingual classroom. In English Pronunciation Instruction [AILA Applied Linguistics Series, 19], ► pp. 39 ff.
Barbier, Guillaume, Ryme Merzouki, Mathilde Bal, Shari R. Baum & Douglas M. Shiller
Bu, Yaohua, Tianyi Ma, Weijun Li, Hang Zhou, Jia Jia, Shengqi Chen, Kaiyuan Xu, Dachuan Shi, Haozhe Wu, Zhihan Yang, Kun Li, Zhiyong Wu, Yuanchun Shi, Xiaobo Lu & Ziwei Liu
Mozaffari, M. Hamed & Won-Sook Lee
Cibelli, Emily
Ciriza, María del Puy & Ahmed Rivera-Campos
Lima, Edna F.
Roon, Kevin D., Jaekoo Kang & D.H. Whalen
Chun, Dorothy M.
Hardison, Debra M.
Hardison, Debra M.
Klaus, Adam, Daniel R. Lametti, Douglas M. Shiller & Tara McAllister
Li, Joanne Jingwen, Samantha Ayala, Daphna Harel, Douglas M. Shiller & Tara McAllister
Terband, Hayo, Aravind Namasivayam, Edwin Maas, Frits van Brenk, Marja-Liisa Mailend, Sanne Diepeveen, Pascal van Lieshout & Ben Maassen
Johnson, Khia Anne, Gloria Madeleine Mellesmoen, Roger Yu-Hsiang Lo & Bryan Gick
Levis, John M.
Levis, John M.
2018. Technology and second language pronunciation. Journal of Second Language Pronunciation 4:2 ► pp. 173 ff.
Levis, John M.
2024. Key issues in L2 pronunciation research. Journal of Second Language Pronunciation 10:3 ► pp. 293 ff.
This list is based on CrossRef data as of 13 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
