Cover not available

In:Multifaceted Multilingualism
Edited by Kleanthes K. Grohmann
[Studies in Bilingualism 66] 2024
► pp. 232–253

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 9
Multimodal story-retelling

Influences of cognitive load on co‑speech and co‑thought gestures for conceptualization

 | National Taipei University of Technology
 | National Taipei University of Technology
Published online: 4 April 2024
DOI logo with link
https://doi.org/10.1075/sibil.66.09yan
Abstract
Cognitive load has been shown to influence speakers’ gesture production, but how it effects the duration, form, and functions of gestures is not yet well understood. This study examines how increased cognitive load influenced the production of co‑speech and co‑thought gestures by second language (L2) learners, and further looks at how different types of gestures correspond to specific functions. Twenty-four English as an L2 participants were recruited and required to watch a pre-determined cartoon and then retell the story in English. Participants were divided into two cognitive load conditions for the story retelling: one group was given a task structure for story retelling, while the other was not. The results of this study show that enhancing cognitive load significantly increased the frequency of both co‑speech and co‑thought gestures and the duration of co‑speech gestures, but not co‑thought gestures. With regard to the correspondence of gesture types and their functions for conceptualization (i.e., activation, manipulation, packaging, exploration), this study found co‑speech and co‑thought gestures serve different functions in L2 narrative tasks. Activation is dominant in every type of co‑speech gesture, especially in iconic and metaphoric gestures. Deictic gestures mostly function for packaging and exploration and co‑thought gestures for exploration and manipulation. This study provides necessary clarification on the relationship between cognitive load and gestures, including novel interactions of gesture duration and functions of gestures for conceptualization.
Article outline
  • 1.Introduction
    • 1.1Types of gestures
    • 1.2Cognitive load and gestures
    • 1.3Information Packaging Hypothesis
    • 1.4Gesture-for-Conceptualization Hypothesis
  • 2.Methodology
    • 2.1Participants and stimuli
    • 2.2Tasks
    • 2.3Gesture coding
  • 3.Results and discussion
    • 3.1Cognitive load and frequency of gestures
    • 3.2Cognitive load and duration of gestures
    • 3.3Functions of co‑speech and co‑thought gestures
  • 4.Conclusion
  • References
  • Appendix
References (49)
References
Alibali, M. W., Spencer, R. C., Knox, L., & Kita, S. (2011). Spontaneous gestures influence strategy choices in problem solving. Psychological Science, 22(9), 1138–1144. Google Scholar logo with link to Google Scholar
Alibali, M. W., Kita, S., & Young, A. J. (2000). Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes, 15(6), 593–613. Google Scholar logo with link to Google Scholar
Anthony, L., Carrington, P., Chu, P., Kidd, C., Lai, J., & Sears, A. (2011). Gesture dynamics: Features sensitive to task difficulty and correlated with physiological sensors. Stress, 1418(360), 312–316.Google Scholar logo with link to Google Scholar
Bava Harji, M., & Gheitanchian, M. (2017). Effects of multimedia task-based teaching and learning approach on EFL learners’ accuracy, fluency and complexity of oral production. Turkish Online Journal of Educational Technology-TOJET, 16(2), 25–34.Google Scholar logo with link to Google Scholar
Butcher, C., & Goldin-Meadow, S. (2000). Gesture and the transition from one- to two-word speech: When hand and mouth come together. In D. McNeill (Ed.), Language and gesture (pp. 235–257). Cambridge University Press. Google Scholar logo with link to Google Scholar
Chen, F., Ruiz, N., Choi, E., Epps, J., Khawaja, M. A., Taib, R., & Wang, Y. (2012). Multimodal behavior and interaction as indicators of cognitive load. ACM Transactions on Interactive Intelligent Systems (TiiS), 2(4), 22. Google Scholar logo with link to Google Scholar
Chu, M., & Kita, S. (2011). The nature of gestures’ beneficial role in spatial problem solving. Journal of Experimental Psychology: General, 140(1), 102. Google Scholar logo with link to Google Scholar
(2016). co‑thought and co‑speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(2), 257.Google Scholar logo with link to Google Scholar
Cook, S. W., Yip, T. K., & Goldin-Meadow, S. (2012). Gestures, but not meaningless movements, lighten working memory load when explaining math. Language and Cognitive Processes, 27(4), 594–610. Google Scholar logo with link to Google Scholar
Cornoldi, C., & Vecchi, T. (2003). Visuo-spatial working memory and individual differences. Psychology Press.Google Scholar logo with link to Google Scholar
Eielts, C., Pouw, W., Ouwehand, K., Van Gog, T., Zwaan, R. A., & Paas, F. (2018). co‑thought gesturing supports more complex problem solving in subjects with lower visual working-memory capacity. Psychological Research, 84, 502–513. Google Scholar logo with link to Google Scholar
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesturing lightens the load. Psychological Science, 12(6), 516–522. Google Scholar logo with link to Google Scholar
Hostetter, A. B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137(2), 297. Google Scholar logo with link to Google Scholar
Hostetter, A. B., Alibali, M. W., & Kita, S. (2007). I see it in my hands’ eye: Representational gestures reflect conceptual demands. Language and Cognitive Processes, 22(3), 313–336. Google Scholar logo with link to Google Scholar
Hostetter, A. B., & Alibali, M. W. (2019). Gesture as simulated action: Revisiting the framework. Psychonomic Bulletin & Review, 26(3), 721–752. Google Scholar logo with link to Google Scholar
Hussain, M. (2014). Cross linguistic variation in the gestures accompanying manner of motion event descriptions by native speakers of English and Urdu. Balochistan Journal of Linguistics, 2, 139–167.Google Scholar logo with link to Google Scholar
Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press. Google Scholar logo with link to Google Scholar
Khatib, M., & Farahanynia, M. (2020). Planning conditions (strategic planning, task repetition, and joint planning), cognitive task complexity, and task type: Effects on L2 oral performance. System, 93, 102297. Google Scholar logo with link to Google Scholar
Kita, S. (2000). How representational gestures help speaking. In D. McNeil (Ed.), Language and gesture (pp. 162–185). Cambridge University Press. Google Scholar logo with link to Google Scholar
Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48(1), 16–32. Google Scholar logo with link to Google Scholar
Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245. Google Scholar logo with link to Google Scholar
Krauss, R. M., Chen, Y., & Gottesman, R. F. (2000) Lexical gestures and lexical access: A process model. In D. McNeill (Ed.), Language and gesture (pp. 261–283). Cambridge University Press. Google Scholar logo with link to Google Scholar
Lee, H. H. (2017). Linguistic-gestural representations in motion event narrations: Mandarin Chinese as L1 and English as L2 (Unpublished Master’s thesis). National Taipei University of Technology.Google Scholar logo with link to Google Scholar
Lee, H., Hampel, R., & Kukulska-Hulme, A. (2019). Gesture in speaking tasks beyond the classroom: An exploration of the multimodal negotiation of meaning via Skype videoconferencing on mobile devices. System, 81, 26–38. Google Scholar logo with link to Google Scholar
Lin, Y. L. (2017). Co-occurrence of speech and gestures: A multimodal corpus linguistic approach to intercultural interaction. Journal of Pragmatics, 117, 155–167. Google Scholar logo with link to Google Scholar
(2022). Speech-accompanying gestures in L1 and L2 conversational interaction by speakers of different proficiency levels. International Review of Applied Linguistics in Language Teaching, 60(2), 123–142.Google Scholar logo with link to Google Scholar
(2020). A helping hand for thinking and speaking: Effects of gestures and task planning on second language narrative discourse, System, 91: 102243. Google Scholar logo with link to Google Scholar
(2021). Gestures as scaffolding for L2 narrative recall: The role of gesture type, task complexity, and working memory. Language Teaching Research. Google Scholar logo with link to Google Scholar
Logan, T., Lowrie, T., & Diezmann, C. M. (2014). co‑thought gestures: Supporting students to successfully navigate map tasks. Educational Studies in Mathematics, 87(1), 87–102. Google Scholar logo with link to Google Scholar
Lopez-Ozieblo, R. (2022). Cut-offs and co-occurring gestures: Similarities between speakers’ first and second languages. International Review of Applied Linguistics in Language Teaching. Google Scholar logo with link to Google Scholar
McNeill, D. (1992). Hand and mind: What gestures reveal about thought. The University of Chicago Press.Google Scholar logo with link to Google Scholar
(2005). Gesture and thought. The University of Chicago Press. Google Scholar logo with link to Google Scholar
(2016). Why we gesture: The surprising role of hand movement in communication. Cambridge University Press.Google Scholar logo with link to Google Scholar
McNeill, D., & Duncan, S. (2000). Growth points in thinking-for-speaking. In D. McNeill (Ed.). Language and gesture (pp. 141–161). Cambridge University Press.Google Scholar logo with link to Google Scholar
Mehrang, F., & Rahimpour, M. (2010). The impact of task structure and planning conditions on oral performance of EFL learners. Procedia-Social and Behavioral Sciences, 2(2), 3678–3686. Google Scholar logo with link to Google Scholar
Melinger, A., & Kita, S. (2007). Conceptualisation load triggers gesture production. Language and Cognitive Processes, 22(4), 473–500. Google Scholar logo with link to Google Scholar
Ping, R., & Goldin-Meadow, S. (2010). Gesturing saves cognitive resources when talking about nonpresent objects. Cognitive Science, 34(4), 602–619. Google Scholar logo with link to Google Scholar
Nakatsukasa, K. (2021). Gesture-enhanced recasts have limited effects: A case of the regular past tense. Language Teaching Research, 25, 587–612. Google Scholar logo with link to Google Scholar
Nicoladis, E., Pika, S., Yin, H. U. I., & Marentette, P. (2007). Gesture use in story recall by Chinese – English bilinguals. Applied Psycholinguistics, 28(4), 721–735. Google Scholar logo with link to Google Scholar
Robinson, P. (2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), 27–57. Google Scholar logo with link to Google Scholar
(2005). Cognitive complexity and task sequencing: A review of studies in a componential framework for second language task design. International Review of Applied Linguistics in Language Teaching, 43(1), 1–33. Google Scholar logo with link to Google Scholar
(2007). Criteria for classifying and sequencing pedagogic tasks. In M. d. P. Garcia Mayo (Ed.), Investigating tasks in formal language learning (pp. 7–27). Multilingual Matters.Google Scholar logo with link to Google Scholar
Sassenberg, U., & Van Der Meer, E. (2010). Do we really gesture more when it is more difficult? Cognitive Science, 34(4), 643–664. Google Scholar logo with link to Google Scholar
Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press.Google Scholar logo with link to Google Scholar
Skehan, P., & Foster, P. (1999). The influence of task structure and processing conditions on narrative retellings. Language Learning, 49(1), 93–120. Google Scholar logo with link to Google Scholar
Smithson, L., & Nicoladis, E. (2014). Lending a hand to imagery? The impact of visuospatial working memory interference upon iconic gesture production in a narrative task. Journal of Nonverbal Behavior, 38(2), 247–258. Google Scholar logo with link to Google Scholar
Talmy, L. (2000). Toward a cognitive semantics (Vol. 2.). The MIT Press.Google Scholar logo with link to Google Scholar
Wagner, S. M., Nusbaum, H., & Goldin-Meadow, S. (2004). Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language, 50(4), 395–407. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue