Cover not available

Article published In: Perspectives on Chinese Language Acquisition
Edited by Henghua Su
[International Journal of Chinese Linguistics 12:2] 2025
► pp. 256–285

Get fulltext from our e-platform
JBE logo with link
EPUB logo with download linkDownload EPUB
authorization required

Word learning and motivation through multimodal input for novice Chinese language learners

 | University at Albany, SUNY
 | University at Albany, SUNY
Published online: 6 October 2025
DOI logo with link
https://doi.org/10.1075/ijchl.00039.hua
Abstract
Non-cognitive factors, such as language input delivered through multimodal channels, may influence second and foreign language learning performance and learner motivation. A substantial body of research has demonstrated the facilitative role of visual aids, such as pictures and videos (Ramezanali, N., & Faez, F. (2019). Vocabulary learning and retention through multimedia glossing. Language Learning and Technology, 23(2), 105–124. Google Scholar logo with link to Google Scholar), as well as bodily movements and gestures (Macedonia, M., & Knösche, T. R. (2011). Body in mind: How gestures empower foreign language learning. Mind, Brain, and Education, 5(4), 196–211. Google Scholar logo with link to Google Scholar), in vocabulary learning. These studies on language embodiment also highlight learners’ general preference for multimodal instructional materials (Lin, C. C., & Yu, Y. C. (2017). Effects of presentation modes on mobile-assisted vocabulary learning and cognitive load. Interactive Learning Environments, 25(4), 528–542. Google Scholar logo with link to Google Scholar). However, most existing research has focused on languages with alphabetic scripts, leaving open questions about how multimodal input affects learning outcomes and motivation in languages with non-alphabetic writing systems. This study addresses this gap by investigating Mandarin Chinese, one of the most widely spoken languages in the world with a logographic writing system, among adult novice learners in three instructional conditions: a traditional pinyin-only group, a picture-enriched group, and a gesture-enriched group. Preliminary findings from a sample of 20 participants reveal that participants in the picture-enriched (n = 6) demonstrated significantly greater gains in motivation from pre- to post-learning compared to those in the pinyin-only group (n = 8) (p < .05), with the gesture-enriched group (n = 6) showing marginal gains (p = .08). However, there are no significant differences across three learning conditions in word learning outcomes. These results align with previous findings on the motivational benefits of multimodal instruction and contribute to our understanding of how such approaches might be effectively integrated into L2 pedagogy for logographic scripts.
Article outline
  • 1.Introduction
    • 1.1Chinese as a Foreign Language (CFL) education
    • 1.2Embodied cognition and the use of multimodal materials in second language (L2) learning
    • 1.3Comparison between picture- and gesture-enriched L2 learning
    • 1.4Motivation in second language learning and the role of multimodal materials
    • 1.5This current study
  • 2.Method
    • 2.1Participants and sampling procedure
    • 2.2Materials and design
      • 2.2.1Chinese word learning effectiveness
        • Stimuli
        • Learning
        • Testing
      • 2.2.2Motivation
      • 2.2.3Chinese phonetic awareness
      • 2.2.4Working memory
    • 2.3Procedure
  • 3.Results
    • 3.1Chinese word learning effectiveness
    • 3.2Motivation change
  • 4.Discussion
    • 4.1Multimodal materials in CFL classroom
    • 4.2Motivation and multimodal materials as influential factors of L2 learning success
      • Educational implications
    • 4.3Limitation and next steps
  • 5.Conclusion
  • References
References (62)
References
Andrä, C., Mathias, B., Schwager, A., Macedonia, M., & von Kriegstein, K. (2020). Learning foreign language vocabulary with gestures and pictures enhances vocabulary memory for several months post-learning in eight-year-old school children. Educational Psychology Review 321, 815–850. Google Scholar logo with link to Google Scholar
Blanco, C. (2024, December 1). 2024 Duolingo language report: What languages came out on top in 2024? Can you guess what’s changed? Duolingo Blog. [URL]
Chan, J., Woore, R., Molway, L., & Mutton, T. (2022). Learning and teaching Chinese as a foreign language: A scoping review. Review of Education, 10(3), e3370. Google Scholar logo with link to Google Scholar
Colenbrander, D., Miles, K. P., & Ricketts, J. (2019). To see or not to see: How does seeing spellings support vocabulary learning?. Language, Speech, and Hearing Services in Schools, 50(4), 609–628. Google Scholar logo with link to Google Scholar
Deci, E. L., Eghrari, H., Patrick, B. C., & Leone, D. R. (1994). Facilitating internalization: The self-determination theory perspective. Journal of personality, 62(1), 119–142. Google Scholar logo with link to Google Scholar
de Groot, A. M., & Keijzer, R. (2000). What is hard to learn is easy to forget: The roles of word concreteness, cognate status, and word frequency in foreign-language vocabulary learning and forgetting. Language Learning, 50(1), 1–56. Google Scholar logo with link to Google Scholar
Dewaele, J.-M., & Alfawzan, M. (2018). Does the effect of enjoyment outweigh that of anxiety in foreign language performance? Studies in Second Language Learning and Teaching, 81, 21–25. Google Scholar logo with link to Google Scholar
Dewaele, J. M., & MacIntyre, P. D. (2014). The two faces of Janus? Anxiety and enjoyment in the foreign language classroom. Studies in second language learning and teaching, 4(2), 237–274. Google Scholar logo with link to Google Scholar
Dewaele, J.-M., & Proietti Ergün, A. L. (2020). How different are the relations between enjoyment, anxiety, attitudes/motivation and course marks in pupils’ Italian and English as foreign languages? Journal of the European Second Language Association, 4(1), 45–57. Google Scholar logo with link to Google Scholar
Dörnyei, Z. (1998). Motivation in second and foreign language learning. Language Teaching, 31(3), 117–135. Google Scholar logo with link to Google Scholar
Dressman, D. (2019). Multimodality and language learning. In M. Dressman & R. W. Sadler (Eds.), The handbook of informal language learning (pp. 39–55). Wiley-Blackwell. Google Scholar logo with link to Google Scholar
Er, S. (2013). Using total physical response method in early childhood foreign language teaching environments. Procedia-Social and Behavioral Sciences, 931, 1766–1768. Google Scholar logo with link to Google Scholar
Everson, M. E. (1998). Word recognition among learners of Chinese as a foreign language: Investigating the relationship between naming and knowing. The Modern Language Journal, 82(2), 194–204. Google Scholar logo with link to Google Scholar
Gardner, R. C. (2004). Attitudes/Motivation Test Battery: International AMTB research project (English version). Canada: The University of Western Ontario. [URL]
International Center for Language Studies (ICLS). (2024, August 20). Most spoken languages in the world. International Center for Language Studies. [URL]
Jaeggi, S. M., Buschkuehl, M., Perrig, W. J., & Meier, B. (2010). The concurrent validity of the N-back task as a working memory measure. Memory, 18(4), 394–412. Google Scholar logo with link to Google Scholar
Jin, Y., & Zhang, L. J. (2018). The dimensions of foreign classroom enjoyment and their effect on foreign language achievement. International Journal of Bilingual Education and Bilingualism, 24(7), 948–962. Google Scholar logo with link to Google Scholar
Lee, W. J., Huang, C. W., Wu, C. J., Huang, S. T., & Chen, G. D. (2012, July). The effects of using embodied interactions to improve learning performance [Conference presentation]. 2012 IEEE 12th International Conference on Advanced Learning Technologies, Rome, Italy.
Li, W., Yu, J., Zhang, Z., & Liu, X. (2022). Dual coding or cognitive load? Exploring the effect of multimodal input on English as a foreign language learners’ vocabulary learning. Frontiers in Psychology, 131, 834706. Google Scholar logo with link to Google Scholar
Lin, C. (2018). Motivation in U.S learners of Mandarin as a foreign and heritage language [Doctoral dissertation, University of Hawaiʻi at Mānoa]. ScholarSpace.
Lin, C. C., & Yu, Y. C. (2017). Effects of presentation modes on mobile-assisted vocabulary learning and cognitive load. Interactive Learning Environments, 25(4), 528–542. Google Scholar logo with link to Google Scholar
Lin, D., Chen, G., Liu, Y., Liu, J., Pan, J., & Mo, L. (2018). Tracking the eye movement of four years old children learning Chinese words. Journal of Psycholinguistic Research, 47(1), 79–93. Google Scholar logo with link to Google Scholar
Lin, Y.-J., & Wang, H.-C. (2021). Using virtual reality to facilitate learners’ creative self-efficacy and intrinsic motivation in an EFL classroom. Education and Information Technologies, 26(4), 4487–4505. Google Scholar logo with link to Google Scholar
Liu, H. C. (2021). Using eye-tracking technology to explore the impact of instructional multimedia on CFL learners’ Chinese character recognition. The Asia-Pacific Education Researcher, 30(1), 33–46. Google Scholar logo with link to Google Scholar
Lu, M. (2011, October 14–16). Chinese character learning: Using embodiment in initial stage [Conference presentation]. American Association for Chinese Studies Conference 2011, Philadelphia, PA, United States. [URL]
Macedonia, M., & Knösche, T. R. (2011). Body in mind: How gestures empower foreign language learning. Mind, Brain, and Education, 5(4), 196–211. Google Scholar logo with link to Google Scholar
Martín-Luengo, B., Hu, Z., Cadavid, S., & Luna, K. (2023). Do pictures influence memory and metamemory in Chinese vocabulary learning? Evidence from Russian and Colombian learners. Plos One, 18(11), e0286824. Google Scholar logo with link to Google Scholar
Martin, K. I., & Ellis, N. C. (2012). The roles of phonological short-term memory and working memory in L2 grammar and vocabulary learning. Studies in Second Language Acquisition, 34(3), 379–413. Google Scholar logo with link to Google Scholar
Mathias, B., Andrä, C., Schwager, A., Macedonia, M., & von Kriegstein, K. (2022). Twelve-and fourteen-year-old school children differentially benefit from sensorimotor-and multisensory-enriched vocabulary training. Educational Psychology Review, 34(3), 1739–1770. Google Scholar logo with link to Google Scholar
Mayer, K. M., Yildiz, I. B., Macedonia, M., & von Kriegstein, K. (2015). Visual and motor cortices differentially support the translation of foreign language words. Current Biology, 25(4), 530–535. Google Scholar logo with link to Google Scholar
McBride, C. A. (2016). Is Chinese special? Four aspects of Chinese literacy acquisition that might distinguish learning Chinese from learning alphabetic orthographies. Educational Psychology Review, 281, 523–549. Google Scholar logo with link to Google Scholar
Nathan, M. J., Walkington, C., Boncoddo, R., Pier, E., Williams, C. C., & Alibali, M. W. (2014). Actions speak louder with words: The roles of action and pedagogical language for grounding mathematical proof. Learning and Instruction, 331, 182–193. Google Scholar logo with link to Google Scholar
Nuraeni, C. (2019). Using total physical response (TPR) method on young learners English language teaching. Metathesis: Journal of English Language, Literature, and Teaching, 3(1), 26–34. Google Scholar logo with link to Google Scholar
Olmanson, J., & Liu, X. C. (2017). The challenge of Chinese character acquisition: Leveraging multimodality in overcoming a centuries-old problem. Emerging Learning Design Journal, 4(1), 1–9. [URL]
Olmanson, J., Liu, X., Heselton, C. C., & Srivastava, A. (2019). DaZiBao: Multimodality in learning and communicating via Chinese characters. In K. -Y. Sung (Ed.), Teaching and Learning Chinese as a Second or Foreign Language: Emerging Trends (pp. 189–203). Rowman & Littlefield. Google Scholar logo with link to Google Scholar
Opitz, B., Schneiders, J. A., Krick, C. M., & Mecklinger, A. (2014). Selective transfer of visual working memory training on Chinese character learning. Neuropsychologia, 531, 1–11. Google Scholar logo with link to Google Scholar
Paivio, A. (1991). Dual coding theory: Retrospect and current status. Canadian Journal of Psychology/Revue Canadienne de Psychologie, 45(3), 255–287. Google Scholar logo with link to Google Scholar
Papi, M. (2010). The L2 motivational self system, L2 anxiety, and motivated behavior: A structural equation modeling approach. System, 38(3), 467–479. Google Scholar logo with link to Google Scholar
Peng, J. E., & MacIntyre, P. D. (2025). Directional or reciprocal? A random intercept cross-lagged panel analysis of the relationships between emotions, motivation, willingness to communicate, and L2 achievement. Learning and Individual Differences, 1181, 102638. Google Scholar logo with link to Google Scholar
Perfetti, C. A., Liu, Y., & Tan, L. H. (2005). The lexical constituency model: Some implications of research on Chinese for general theories of reading. Psychological Review, 112(1), 43–59. Google Scholar logo with link to Google Scholar
Quak, M., London, R. E., & Talsma, D. (2015). A multisensory perspective of working memory. Frontiers in Human Neuroscience, 91, 197. Google Scholar logo with link to Google Scholar
Ramezanali, N., & Faez, F. (2019). Vocabulary learning and retention through multimedia glossing. Language Learning and Technology, 23(2), 105–124. Google Scholar logo with link to Google Scholar
Repetto, C., Pedroli, E., & Macedonia, M. (2017). Enrichment effects of gestures and pictures on abstract words in a second language. Frontiers in Psychology, 81, 2136. Google Scholar logo with link to Google Scholar
Ryan, R. M. (1982). Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology, 43(3), 450–461. Google Scholar logo with link to Google Scholar
Sadoski, M. (2018). Reading comprehension is embodied: Theoretical and practical considerations. Educational Psychological Review, 301, 331–349. Google Scholar logo with link to Google Scholar
Saito, K., Dewaele, J. M., Abe, M., & In’nami, Y. (2018). Motivation, emotion, learning experience, and second language comprehensibility development in classroom settings: A cross-sectional and longitudinal study. Language Learning, 68(3), 709–743. Google Scholar logo with link to Google Scholar
Shu, H., Peng, H., & McBride-Chang, C. (2008). Phonological awareness in young Chinese children. Developmental Science, 11(1), 171–181. Google Scholar logo with link to Google Scholar
Sorden, S. D. (2012). The cognitive theory of multimedia learning. In B. J. Irby, G. Brown, R. Lara-Alecio, & S. Jackson (Eds.), Handbook of educational theories (pp. 155–165). Information Age Publishing, Inc.Google Scholar logo with link to Google Scholar
Sun, H., Wang, T., & Yu, M. (2024). Influence of stroke-order learning on Chinese character recognition among Chinese as a second language learners. System, 1221, Google Scholar logo with link to Google Scholar
Teng, M. F. (2023). The effectiveness of multimedia input on vocabulary learning and retention. Innovation in Language Learning and Teaching, 17(3), 738–754. Google Scholar logo with link to Google Scholar
Teng, M., & Xu, J. (2025). Pushing vocabulary knowledge from receptive to productive mastery: Effects of task type and repetition frequency. Language Teaching Research, 29(2), 588–606. Google Scholar logo with link to Google Scholar
Tseng, C. C., Hu, J. F., Chang, L. Y., & Chen, H. C. (2023). Learning to read Chinese: The roles of phonological awareness, paired–associate learning, and phonetic radical awareness. Reading and Writing, 36(7), 1769–1795. Google Scholar logo with link to Google Scholar
Verhagen, J., & Leseman, P. (2016). How do verbal short-term memory and working memory relate to the acquisition of vocabulary and grammar? A comparison between first and second language learners. Journal of Experimental Child Psychology, 1411, 65–82. Google Scholar logo with link to Google Scholar
von Kriegstein, K. (2012). A multisensory perspective on human auditory communication. In M. M. Murray & M. T. Wallace (Eds.), The neural bases of multisensory processes (pp. 683–700). Boca Raton: Taylor & FrancisGoogle Scholar logo with link to Google Scholar
Wu, C.-J., Chen, G.-D., & Huang, C.-W. (2014). Using digital board games for genuine communication in EFL classrooms. Educational Technology Research and Development, 62(2), 209–226. Google Scholar logo with link to Google Scholar
Xiao, W., & Treiman, R. (2012). Iconicity of simple Chinese characters. Behavior Research Methods, 441, 954–960. Google Scholar logo with link to Google Scholar
Xu, S. (100 AD). Shuowen jiezi [Explaining simple and analyzing compound characters]. [URL]
Xu, X., & Ke, F. (2020). Embodied interaction: Learning Chinese characters through body movements. Language Learning & Technology, 24(3), 136–159. [URL].
Xu, Y., Chang, L. Y., Zhang, J., & Perfetti, C. A. (2013). Reading, writing, and animation in character learning in Chinese as a foreign language. Foreign Language Annals, 46(3), 423–444. Google Scholar logo with link to Google Scholar
Yang, Y., Leung, H., Shum, H. P., Li, J., Zeng, L., Aslam, N., & Pan, Z. (2017). CCESK: A Chinese character educational system based on Kinect. IEEE Transactions on Learning Technologies, 11(3), 342–347. Google Scholar logo with link to Google Scholar
Yashima, T., Nishida, R., & Mizumoto, A. (2017). Influence of learner beliefs and gender on the motivating power of L2 selves. The Modern Language Journal, 101(4), 691–711. Google Scholar logo with link to Google Scholar
Zou, L., Desroches, A. S., Liu, Y., Xia, Z., & Shu, H. (2012). Orthographic facilitation in Chinese spoken word recognition: An ERP study. Brain and Language, 123(3), 164–173. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue