Cover not available

Article published In: Gesture
Vol. 20:3 (2021) ► pp.354–375

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

High verbal working memory load impairs gesture-speech integration

Evidence from a dual task paradigm

 | University of Mons
 | University of Hull
 | University of Mons
 | University of Mons
 | University of Mons
Published online: 4 August 2022
DOI logo with link
https://doi.org/10.1075/gest.20028.kan
Abstract
While previous studies have shown the importance of visuo-spatial working memory in the processing of co-speech iconic gestures, clear evidence for a potential involvement of the verbal working memory (vWM) is currently lacking. To address this issue, participants in the present study were presented with a dual task paradigm. The main outcome variable was the performance on a Stroop-like gesture task which provides a behavioural index of gesture-speech integration. Participants performed this task under conditions of either high or low concurrent vWM load. Unlike in previous studies, the number of words to remember in the high load condition was determined by their individual verbal span rather than being fixed. Results showed reaction time costs in the form of longer reaction times for semantically incongruent gesture-speech combinations as compared to congruent combinations. However, this semantic congruency effect disappeared when the vWM load increased. This result suggests a causal involvement of verbal working memory capacity in gesture-speech integration.
Article outline
  • Introduction
  • Methods
    • Participants
    • Materials
      • Digit Span task
      • Dual-task paradigm
    • Experimental design and statistical analysis
  • Results
    • Main analysis
      • Gender Classification Task
      • Secondary verbal working memory task
  • Discussion
  • Conclusion
  • References
References (56)
References
Alibali, Martha W. (2005). Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information. Spatial Cognition and Computation, 5 (4), 307–331. Google Scholar logo with link to Google Scholar
Alibali, Martha. W., Flevares, Lucia M., & Goldin-Meadow, Susan. (1997). Assessing knowledge conveyed in gesture: Do teachers have the upper hand? Journal of Educational Psychology, 89 (1), 183–193. Google Scholar logo with link to Google Scholar
Baddeley, Alan D. (1992). Working memory. Science, 225(5044), 556–559. Google Scholar logo with link to Google Scholar
(2012). Working memory: Theories, Models and Controversies. Annual Reviews of Psychology, 631, 1–29. Google Scholar logo with link to Google Scholar
Baddeley, Alan D., & Hitch, Graham J. (1974). Working memory. In Gordon H. Bower (Ed.), The psychology of learning and motivation: Advances in research and theory (Vol. 81, pp. 47–89). London: Academic Press.Google Scholar logo with link to Google Scholar
Beattie, Geoffrey & Shovelton, Heather. (2002). What properties of talk are associated with the generation of spontaneous hand gestures? British Journal of Social Psychology, 411, 403–417. Google Scholar logo with link to Google Scholar
Besner, Derek & Stolz, Jennifer A. (1999). Unconsciously controlled processing: The Stroop effect reconsidered. Psychonomic Bulletin & Review, 6 (3), 449–455. Google Scholar logo with link to Google Scholar
Boehringer, Andreas, Macher, Katja, Dukart, Juergen, Villringer, Arno, & Pleger, Burkhard. (2013). Cerebellar transcranial direct stimulation modulates verbal working memory. Brain Stimulation, 61, 649–653. Google Scholar logo with link to Google Scholar
Chu, Minguyan, Meyer, Antje, Foulkes, Lucy, & Kita, Sotaro. (2014). Individual differences in frequency and saliency of speech-accompanying gestures: The role of cognitive abilities and empathy. Journal of Experimental Psychology: General, 143 (2), 694–709. Google Scholar logo with link to Google Scholar
Daneman, Meredith & Carpenter, Patricia A. (1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behavior, 191, 450–466. Google Scholar logo with link to Google Scholar
Delaloye, Christophe, Ludwig, Catherine, Borella, Erika, Chicherio, Christian, & de Ribaupierre, Anik. (2008). The reading span as a measure of working memory capacity: Norms based on a French speaking population of 775 younger and older adults. Revue européenne de Psychologie Appliquée, 581, 89–103. Google Scholar logo with link to Google Scholar
Dick, Anthony, Goldin-Meadow, Susan, Hasson, Uri, Skipper, Jeremy, & Small, Steven L. (2009). Co-speech gestures influence neural activity in brain regions associated with processing semantic information. Human Brain Mapping, 301, 3509–3526. Google Scholar logo with link to Google Scholar
Engel, Randall W. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11 (1), 19–23. Google Scholar logo with link to Google Scholar
Feyereisen, Pierre & Havard, Isabelle. (1999). Mental imagery and production of hand gestures while speaking in younger and older adults. Journal of Non-Verbal Behavior, 231, 153–171. Google Scholar logo with link to Google Scholar
Gauthier, Janel & Bouchard, Stéphane. (1993). A French-Canadian adaptation of the revised version of Spielberger’s State-Trait Anxiety Inventory. Canadian Journal of Behavioural Science, 25 (4), 559–578. Google Scholar logo with link to Google Scholar
Gillepsie, Maureen, James, Ariel N., Federmeier, Kara D., & Watson, Duane G. (2014). Verbal working memory predicts co-speech gesture: Evidence from individual differences. Cognition, 1321, 174–180. Google Scholar logo with link to Google Scholar
Green, Antonia, Straube, Benjamin, Weis, Susanne, Jansen, Andreas, Wilmes, Klaus, Konrad, Kerstin, & Kircher, Tilo. (2009). Neural integration of iconic and unrelated co-verbal gestures: A functional MRI study. Human Brain Mapping, 30 (10), 3309–3324. Google Scholar logo with link to Google Scholar
Habets, Boukje, Kita, Sotaro, Shao, Zeshu, Özyürek, Aslı, & Hagoort, Peter. (2010). The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23 (8), 1845–1854. Google Scholar logo with link to Google Scholar
Hadar, Uri & Butterworth, Brian. (1997). Iconic gestures, imagery and word retrieval in speech. Semiotica, 115 (1/2), 147–172. Google Scholar logo with link to Google Scholar
Hadar, Uri & Pinchas-Zamir, Lian. (2004). The semantic specificity of gesture. Implications for gesture classification and function. Journal of Language and Social Psychology, 23 (2), 204–214. Google Scholar logo with link to Google Scholar
Hadwin, Julie, Brogan, Joanna, & Stevenson, Jim. (2005). State anxiety and working memory in children: A test of processing efficiency theory. Educational Psychology, 25 (4), 379–393. Google Scholar logo with link to Google Scholar
Holle, Henning & Gunter, Thomas C. (2007). The role of iconic gestures in speech disambiguation: ERP evidence. Journal of Cognitive Neuroscience, 19 (7), 1175–1192. Google Scholar logo with link to Google Scholar
Holle, Henning, Gunter, Thomas C., Rueschemeyer, Shirley, Hennenlotter, Andreas, & Iacoboni, Marco. (2008). Neural correlates of the processing of co-speech gestures. NeuroImage, 39 (4), 2010–2024. Google Scholar logo with link to Google Scholar
Holler, Judith, Shovelton, Heather, & Beattie, Geoffrey. (2009). Do iconic hand gestures really contribute to the communication of semantic information in face-to-face context? Journal of Non-Verbal Behavior, 33 (2), 73–88. Google Scholar logo with link to Google Scholar
Hostetter, Autumn B. (2011). When do gestures communicate? A meta-analysis. Psychological Bulletin, 137 (2), 297–315. Google Scholar logo with link to Google Scholar
Kelly, Spencer D., Barr, Dale J., Church, R. Breckinridge, & Lynch, Katheryn. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Langage, 401, 577–592. Google Scholar logo with link to Google Scholar
Kelly, Spencer D., Creigh, Peter, & Bartolotti, James. (2009). Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 22 (4), 683–694. Google Scholar logo with link to Google Scholar
Kelly, Spencer, Kravitz, Corinne, & Hopkins, Michael. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain and Language, 89 (1), 253–260. Google Scholar logo with link to Google Scholar
Kelly, Spencer, Özyürek, Aslı, & Maris, Eric. (2010). Two sides of the same coin: Speech and gesture mutually interact to enhance communication. Psychological Science, 21 (2), 260–267. Google Scholar logo with link to Google Scholar
Kelly, Spencer, Ward, Sarah, Creigh, Peter, & Bartolotti, James. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain and Language, 101 (3), 222–233. Google Scholar logo with link to Google Scholar
Kita, Sotaro. (2000). How representational gestures help speaking. In David McNeill (Ed.), Language and gesture (pp. 162–185). Cambridge, UK: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kita, Sotaro & Özyürek, Aslı. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48 (1), 16–32. Google Scholar logo with link to Google Scholar
Krauss, Robert M., Chen, Yihsiu, & Chawla, Purnima. (1996). Non-verbal behavior and non-verbal communication: What do conversational hand gesture tell us? In M. Zanna (Ed.), Advances in experimental social psychology (pp. 389–450). San Diego: Academic Press.Google Scholar logo with link to Google Scholar
Krauss, Robert M., Morrel-Samuels, Palmer, & Colasante, Christina. (1991). Do conversational hand gestures communicate? Journal of Personality and Social Psychology, 61 (5), 743–754. Google Scholar logo with link to Google Scholar
McNeill, David. (1985). So you think gestures are non-verbal? Psychological Review, 92 (3), 350–371. Google Scholar logo with link to Google Scholar
. (1992). Hand and mind. Chicago: University of Chicago Press.Google Scholar logo with link to Google Scholar
McNeill, David, Cassell, Justine, & McCullough, Karl-Erik. (1994). Communicative effects of speech-mismatched gestures. Research on Language and Social Interaction, 27 (3), 223–237. Google Scholar logo with link to Google Scholar
Momsen, Jacob, Gordon, Jared, Wu, Ying Choon, & Coulson, Seana. (2020). Verbal working memory and co- speech gesture processing. Brain and Cognition, 1461, 105640. Google Scholar logo with link to Google Scholar
Morrel-Samuels, Palmer & Krauss, Robert M. (1992). Word familiarity predicts asynchrony of hand gestures and speech. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18 (3), 615–622.Google Scholar logo with link to Google Scholar
Navon, David & Miller, Jeff. (2002). Queuing or sharing? A critical evaluation of the single- bottleneck notion. Cognitive Psychology, 441, 193–251. Google Scholar logo with link to Google Scholar
Obermeier, Christian, Holle, Henning, & Gunter, Thomas. (2011). What iconic gesture fragments reveal about gesture-speech integration: When synchrony is lost, memory can help. Journal of Cognitive Neuroscience, 23 (7), 1648–1663. Google Scholar logo with link to Google Scholar
Ovroye, Acacia L. & Storm, Benjamin C. (2019). Remembering what was said and done: The activation and facilitation of memory for gesture as a consequence of retrieval. Journal of Experimental Psychology: Learning, Memory and Cognition, 45 (3), 526–534.Google Scholar logo with link to Google Scholar
Owens, Matthews, Stevenson, Jim, Hadwin, Julie A., & Norgate, Roger. (2014). When does anxiety help or hinder test performance? The role of working memory capacity. British Journal of Psychology, 105 (1), 92–101. Google Scholar logo with link to Google Scholar
Özer, Demet & Göksun, Tilbe. (2020a). Gesture use and processing: A review on individual differences in cognitive resources. Frontiers in Psychology, 111, 573555. Google Scholar logo with link to Google Scholar
. (2020b). Visual-spatial and verbal abilities differentially affect processing of gestural vs. spoken expressions. Language, Cognition and Neuroscience, 35 (7), 896–914. Google Scholar logo with link to Google Scholar
Özyürek, Aslı, Willems, Roel M., Kita, Sotaro, & Hagoort, Peter. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. Journal of Cognitive Neuroscience, 191, 605–616. Google Scholar logo with link to Google Scholar
Smyth, Mary M. & Pendleton, Lindsay R. (1990). Space and movement in working memory. The Quarterly Journal of Experimental Psychology, Section A: Human Experimental Psychology, 42 (2), 291–304. Google Scholar logo with link to Google Scholar
Vytal, Katherine E., Cornwell, Brian R., Arkin, Nicole Esther, Letkiewicz, Allison M., & Grillon, Christian. (2013). The complex interaction between anxiety and cognition: insight from spatial and verbal working memory. Frontiers in Human Neuroscience, 71, 93. Google Scholar logo with link to Google Scholar
Wagner, Susan M., Nusbaum, Howard, & Goldin-Meadow, Susan. (2004). Probing the mental representation of gesture: Is handwaving spatial? Journal of Memory and Language, 501, 395–407. Google Scholar logo with link to Google Scholar
Wechsler, David. (2008). Wechsler adult intelligence scale – fourth edition.Google Scholar logo with link to Google Scholar
Wolf, Dhana, Rekittke, Linn-Marlen, Mittelberg, Irene, Klasen, Martin, & Mathiak, Klaus. (2017). Perceived conventionality in co-speech gestures involves the fronto-temporal language network. Frontiers in Human Neuroscience, 111, 573. Google Scholar logo with link to Google Scholar
Wu, Ying Choon & Coulson, Seana. (2007). How iconic gestures enhance communication: An ERP study. Brain & Language, 1011, 234–245. Google Scholar logo with link to Google Scholar
. (2011). Are depictive gestures like pictures? Commonalities and differences in semantic processing. Brain & Language, 1191, 184–195. Google Scholar logo with link to Google Scholar
. (2014). Co-speech iconic gestures and visuo-spatial working memory. Acta Psychologia, 1531, 39–50. Google Scholar logo with link to Google Scholar
. (2015). Iconic gestures facilitate discourse comprehension in individuals with superior immediate memory for body configurations. Psychological Science, 26 (11), 1717–1727. Google Scholar logo with link to Google Scholar
Zhao, Wanying, Riggs, Kevin, Schindler, Igor, & Holle, Henning. (2018). Transcranial Magnetic Stimulation over left inferior frontal and posterior temporal cortex disrupts gesture-speech integration. The Journal of Neuroscience, 38 (8), 1891–1900. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue