In:Why Gesture?: How the hands function in speaking, thinking and communicating
Edited by Ruth Breckinridge Church, Martha W. Alibali and Spencer D. Kelly
[Gesture Studies 7] 2017
► pp. 243–265
Chapter 11Exploring the boundaries of gesture-speech integration during language comprehension
Published online: 26 April 2017
https://doi.org/10.1075/gs.7.12kel
https://doi.org/10.1075/gs.7.12kel
Abstract
The present review explores the integrated relationship between gesture and speech during language comprehension. Taking a broad view, it presents a conceptual framework that approaches the comprehension of gesture and speech along three different dimensions: (1) components of language (semantics, pragmatics, phonetics and syntax), (2) levels of analysis (social, cognitive, and biological) and (3) timeframes of integration (online, moment-to-moment, developmental). The evidence suggests that some linguistic components (e.g., concrete semantic and pragmatic) are deeply connected to gestures, but others (e.g., abstract semantic, syntactic and phonetic) are less so. In this way, the hands help to delineate what aspects of language function as part of the body and what aspects operate independently of it.
Article outline
- General introduction
- Converging methods
- The four linguistic components
- Semantics
- Online and moment-to-moment integration
- Developmental integration
- Further exploration
- Pragmatics
- Moment-to-moment and developmental integration
- Online integration
- Further exploration
- Phonetics
- Online and moment-to-moment integration
- Developmental integration
- Further exploration
- Syntax
- Moment-to-moment and developmental integration
- Online integration
- Further exploration
- Semantics
- Conclusion
Notes References
References (95)
Alibali, Martha W., & Goldin-Meadow, Susan 1993. “Gesture-speech mismatch and mechanisms of learning: What the hands reveal about a child’s state of mind.” Cognitive Psychology 25: 468–523.
Armstrong, David F., & Wilcox, Sherman E. 2007. The Gestural Origin of Language. Oxford; New York: Oxford University Press.
Bailey, April, & Kelly, Spencer D. 2015. “Picture power: Gender versus body language in perceived status.” Journal of Nonverbal Behavior 39: 317–337.
Baron-Cohen, Simon 1989. “Perceptual role taking and protodeclarative pointing in autism.” British Journal of Developmental Psychology 7 (2): 113–127.
Barsalou, Lawrence W. 1999. “Perceptions of perceptual symbols.” Behavioral and Brain Sciences, 22(04), 637–660.
Bates, Elizabeth, & Dick, Fredrick 2002. “Language, gesture, and the developing brain.” Developmental Psychobiology 40 (3): 293–310.
Biau, Emmanuel, & Soto-Faraco, Salvador 2013. “Beat gestures modulate auditory integration in speech perception.” Brain and Language 124 (2): 143–152.
Binder, Jeffrey. R., & Desai, Rutvik H. 2011. “The neurobiology of semantic memory.” Trends in Cognitive Sciences 15 (11): 527–536.
Church, Ruth B. 1999. “Using gesture and speech to capture transitions in learning.” Cognitive Development 14: 313–342.
Church, Ruth B., Garber, Philip, & Rogalsky, Kathryn 2007. “The role of gesture in social communication and memory.” Gesture 7: 137–157.
Clark, Herbert H., & Marshall, Catherine R. 1981. “Definite reference and mutual knowledge.” In Elements of Discourse Understanding, A.K. Joshi, B. Webber, I.A. Sag (eds), 10–63. Cambridge University Press, Cambridge.
Corballis, Michael C. 2003. “From mouth to hand: gesture, speech, and the evolution of right-handedness.” Behavioral and Brain Sciences 26 (02): 199–208.
de Langavant, Laurent C., et al. 2011. “Behavioral and neural correlates of communication via pointing.” PloS one 6 (3): e17719.
de Gelder, Beatrice 2006. “Towards the neurobiology of emotional body language.” Nature Reviews Neuroscience 7 (3): 242–249.
Dingemanse, Mark, Blasi, Damián E., Lupyan, Gary, Christiansen, Morten H., & Monaghan, Padraic 2015. “Arbitrariness, iconicity, and systematicity in language.” Trends in Cognitive Sciences 19: 603–615.
Dodd, Barbara 1979. “Lip reading in infants: Attention to speech presented in-and out-of-synchrony.” Cognitive Psychology 11 (4): 478–484.
Gallese, Vittorio, & Lakoff, George 2005. “The brain's concepts: The role of the sensory-motor system in conceptual knowledge.” Cognitive Neuropsychology 22: 455–479.
Goldin-Meadow, Susan, McClintock, Martha, & Wimsatt, William Unpublished manuscript. “Solving psychological problems in four dimensions: Heuristics for the social and biological sciences.”
Goldin-Meadow, Susan, Alibali, Martha W., & Church, Ruth B. 1993. “Transitions in concept acquisition: Using the hand to read the mind.” Psychological Review 100 (2): 279–297.
Glenberg, Arthur M., & Kaschak, Michael P. 2002. “Grounding language in action.” Psychonomic Bulletin & Review 9 (3): 558–565.
Grice, H. Paul 1975. “Logic and conversation.” In P. Cole and J. L. Morgan (eds), Syntax and semantics, (Vol. 3). New York: Academic Press.
Gullberg, Marianne 2006. “Some reasons for studying gesture and second language acquisition (Hommage à Adam Kendon).” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 103–124.
Gunter, Thomas C., & Bach, Patric 2004. “Communicating hands: ERPs elicited by meaningful symbolic hand postures.” Neuroscience Letters 372 (1): 52–56.
Hagoort, Peter, Hald, Lea, Bastiaansen, Marcel, & Petersson, Karl Magnus 2004. “Integration of word meaning and world knowledge in language comprehension.” Science 304 (5669): 438–441.
Hagoort, Peter 2005. “On Broca, brain, and binding: a new framework.” Trends in Cognitive Sciences 9 (9): 416–423.
Hickok, Gregory 2014. The Myth of Mirror Neurons: The Real Neuroscience of Communication and Cognition. WW Norton & Company.
Hinzman, Lindsay & Kelly, Spencer D. 2013. “Effects of emotional body language on rapid out-group judgments.” Journal of Experimental Social Psychology 49: 152–155.
Hirata, Yukari & Kelly, Spencer D. 2010. “Effects of lips and hands on auditory learning of second-language speech sounds.” Journal of Speech, Language, and Hearing Research 53 (2): 298–310.
Hirata, Yukari, Kelly, Spencer D., Huang, Jessica, & Manansala, Michael 2014. “Effects of hand gestures on auditory learning of second-language vowel length contrasts.” Journal of Speech, Language, and Hearing Research 57 (6): 2090–2101.
Holle, Henning, & Gunter, Thomas C. 2007. “The role of iconic gestures in speech disambiguation: ERP evidence.” Journal of Cognitive Neuroscience 19 (7): 1175–1192.
Holler, Judith, Schubotz, Louise, Kelly, Spence D., Hagoort, Peter, Schuetze, Manuela & Özyürek, Asli 2014. “Social eye gaze modulates processing of speech and co-speech gesture.” Cognition 132: 696–697.
Holler, Judith, Kokal, Idil, Toni, Ivan, Hagoort, Peter, Kelly, Spencer D., & Özyürek, Asli 2014. “Eye'm talking to you: Speakers' gaze direction modulates co-speech gesture processing in the right MTG.” Social Cognitive and Affective Neuroscience 10 (2): 255–261.
Hostetter, Autumn B., & Alibali, Martha W. 2008. “Visible embodiment: Gestures as simulated action.” Psychonomic Bulletin & Review 15 (3): 495–514.
Hostetter, Autumn B. 2011. “When do gestures communicate? A meta-analysis.” Psychological Bulletin 137 (2): 297.
Hubbard, Amy L., Wilson, Stephen M., Callan, Daniel E., & Dapretto, Mirella 2009. “Giving speech a hand: gesture modulates activity in auditory cortex during speech perception.” Human Brain Mapping 30 (3): 1028–1037.
Karmiloff-Smith, Anette 1995. Beyond Modularity: A Developmental Perspective on Cognitive Science. MIT Press.
Kelly, Spencer D., Barr, Dale J., Church, Ruth B., & Lynch, Katheryn 1999. “Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory.” Journal of Memory and Language 40 (4): 577–592.
Kelly, Spencer D., Kravitz, Corinne, & Hopkins, Hopkins 2004. “Neural correlates of bimodal speech and gesture comprehension.” Brain and language 89: 253–260.
Kelly, Spencer D. 2001. “Broadening the units of analysis in communication: Speech and nonverbal behaviours in pragmatic comprehension.” Journal of Child Language 28 (02): 325–349.
Kelly, Spencer D., Iverson, Jana M., Terranova, Joseph, Niego, Julia, Hopkins, Michael, & Goldsmith, Leslie 2002. “Putting language back in the body: Speech and gesture on three time frames.” Developmental Neuropsychology 22 (1): 323–349.
Kelly, Spencer D., Ward, Sarah, Creigh, Peter, & Bartolotti, James 2007. “An intentional stance modulates the integration of gesture and speech during comprehension.” Brain and Language 101: 222–233.
Kelly, Spencer D., Manning, Sarah M., & Rodak, Sabrina 2008. “Gesture gives a hand to language and learning: Perspectives from cognitive neuroscience, developmental psychology and education.” Language and Linguistics Compass 2 (4): 569–588.
Kelly, Spencer D., McDevitt, Tara, & Esch, Megan 2009. “Brief training with co-speech gesture lends a hand to word learning in a foreign language.” Language and Cognitive Processes 24 (2): 313–334.
Kelly, Spencer D., Özyürek, Asli, & Maris, Eric. 2010. “Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension.” Psychological Science 21: 260–267.
Kelly, Spencer D., Creigh, Peter, & Bartolotti, James 2010. “Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing.” Journal of Cognitive Neuroscience 22: 683–694.
Kelly, Spencer D., & Lee, Angela L. 2012. “When actions speak too much louder than words: Hand gestures disrupt word learning when phonetic demands are high.” Language and Cognitive Processes 27 (6): 793–807.
Kelly, Spencer D., Healey, Meghan, Özyürek, Asli, & Holler, Judith 2014. “The processing of speech, gesture and action during language comprehension.” Psychonomic Bulletin & Review 22 (2): 517–523.
Kelly, Spencer D., Hirata, Yukari, Manansala, Michael, & Huang, Jessica 2014. “Exploring the role of hand gestures in learning novel phoneme contrasts and vocabulary in a second language.” Frontiers in Psychology, 5.
Kita, Sotaro, & Özyürek, Asli 2003. “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48 (1): 16–32.
Klaver, Peter, et al. 2005. “Word imageability affects the hippocampus in recognition memory.” Hippocampus 15 (6): 704–712.
Knill, David C., & Pouget, Alexandre 2004. “The Bayesian brain: the role of uncertainty in neural coding and computation.” Trends in Neurosciences 27 (12): 712–719.
Krahmer, Emile, & Swerts, Marc 2007. “The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception.” Journal of Memory and Language 57 (3): 396–414.
Kuhl, Patricia K., & Meltzoff, Andrew N. 1984. “The intermodal representation of speech in infants.” Infant Behavior and Development 7 (3): 361–381.
Kutas, Marta, & Hillyard, Steven A. 1980. “Reading senseless sentences: Brain potentials reflect semantic incongruity.” Science 207 (4427): 203–205.
Lakoff, George, & Johnson, Mark 1999. Philosophy In The Flesh: The Embodied Mind And Its Challenge To Western Thought. Basic Books.
Loehr, Daniel 2007. “Aspects of rhythm in gesture and speech.” Gesture 7 (2): 179–214.
Macedonia, Manuela, Müller, Karsten, & Friederici, Angela D. 2011. “The impact of iconic gestures on foreign language word learning and its neural substrate.” Human Brain Mapping 32 (6): 982–998.
Masumoto, Kouhei, et al. 2006. “Reactivation of physical motor information in the memory of action events.” Brain Research 1101 (1): 102–109.
Mahon, Bradford Z., & Caramazza, Alfonso 2008. “A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content.” Journal of Physiology-Paris 102 (1): 59–70.
McNeill, David 1992. Hand and Mind: What Gestures Reveal About Thought. University of Chicago Press.
Morford, Marolyn, & Goldin-Meadow, Susan 1992. “Comprehension and production of gesture in combination with speech in one-word speakers.” Journal of Child Language 19 (03): 559–580.
Murphy, Catherine M., & Messer, David J. 1977. “Mothers, infants and pointing: A study of a gesture.” Studies in Mother-Infant Interaction: 325–354.
Özyürek, Asli, Willems, Roel, Kita, Sotaro, & Hagoort, Peter 2007. “On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials.” Journal of Cognitive Neuroscience 19 (4): 605–616.
Özyürek, Asli 2014. “Hearing and seeing meaning in speech and gesture: insights from brain and behaviour.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130296.
Perry, Michelle, Church, R. Breckinridge, & Goldin-Meadow, Susan 1992. “Is gesture-speech mismatch a general index of transitional knowledge?” Cognitive Development 7 (1): 109–122.
Post, Lysanne S., Van Gog, Tamara, Paas, Fred, & Zwaan, Rolf A. 2013. “Effects of simultaneously observing and making gestures while studying grammar animations on cognitive load and learning.” Computers in Human Behavior 29 (4): 1450–1455.
Povinelli, Daniel J. 1993. “Reconstructing the evolution of mind.” American Psychologist 48: 493–509.
Pulvermüller, F. 2005. “Brain mechanisms linking language and action.” Nature Reviews Neuroscience 6 (7): 576–582.
Rizzolatti, Giacomo, & Arbib, Michael A. 1998. “Language within our grasp.” Trends in Neurosciences 21 (5): 188–194.
Skipper, Jeremy I. 2014. “Echoes of the spoken past: how auditory cortex hears context during speech perception.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130297.
So, Wing C., Kita, Sotaro, & Goldin-Meadow, Susan 2009. “Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand.” Cognitive Science 33 (1): 115–125.
Stam, Gale 2006. “Thinking for speaking about motion: L1 and L2 speech and gesture.” IRAL-International Review of Applied Linguistics in Language Teaching 44 (2): 145–171.
Straube, Benjamin, Green, Antonia, Jansen, Andreas, Chatterjee, Anjan, & Kircher, Tilo 2010. “Social cues, mentalizing and the neural processing of speech accompanied by gestures.” Neuropsychologia 48 (2): 382–393.
Straube, Benjamin, Green, Antonia, Weis, Susanne, & Chatterjee, Anjan 2009. “Memory effects of speech and gesture binding: cortical and hippocampal activation in relation to subsequent memory performance.” Journal of Cognitive Neuroscience 21 (4): 821–836.
Sumby, Willaim H., & Pollack, Irwin 1954. “Visual contribution to speech intelligibility in noise.” The Journal of the Acoustical Society of America 26 (2): 212–215.
Tenenbaum, Joshua B., Kemp, Charles, Griffiths, Thomas L., & Goodman Noah D. 2011. “How to grow a mind: Statistics, structure, and abstraction.” Science 331: 1279–1285.
Thompson, Laura A., & Massaro, Dominic W. 1986. “Evaluation and integration of speech and pointing gestures during referential understanding.” Journal of Experimental Child Psychology 42 (1): 144–168.
1994. “Children's integration of speech and pointing gestures in comprehension.” Journal of Experimental Child Psychology 57 (3): 327–354.
van Wassenhove, Virgini, Grant, Ken W., & Poeppel, David 2005. “Visual speech speeds up the neural processing of auditory speech.” Proceedings of the National Academy of Sciences of the United States of America 102 (4): 1181–1186.
Willems, Roel M., & Hagoort, Peter 2007. “Neural evidence for the interplay between language, gesture, and action: A review.” Brain and Language 101 (3): 278–289.
Wimsatt, William 1981. “Robustness, reliability and overdetermination.” In Scientific Inquiry and the Social Sciences, M. Brewer and B. Collins (eds), 124–163. San Francisco: Jossey-Bass.
Cited by (17)
Cited by 17 other publications
Hirata, Yukari
Kelly, Spencer D. & Quang‐Anh Ngo Tran
Gullberg, Marianne
Nirme, Jens, Agneta Gulz, Magnus Haake & Marianne Gullberg
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
Arbona, Eléonore, Kilian G. Seeber & Marianne Gullberg
Janzen Ulbricht, Natasha & Shane Lindsay
Mlakar, Izidor, Darinka Verdonik, Simona Majhenič & Matej Rojc
Billot-Vasquez, Kiana, Zhongwen Lian, Yukari Hirata & Spencer D. Kelly
Iizuka, Takehiro, Kimi Nakatsukasa & Aaron Braver
Janzen Ulbricht, Natasha
Levy, Rachel S. & Spencer D. Kelly
van Berkel-van Hoof, Lian, Daan Hermans, Harry Knoors & Ludo Verhoeven
Zheng, Annie, Yukari Hirata & Spencer D. Kelly
This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
