Cover not available

In:Why Gesture?: How the hands function in speaking, thinking and communicating
Edited by Ruth Breckinridge Church, Martha W. Alibali and Spencer D. Kelly
[Gesture Studies 7] 2017
► pp. 59–75

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 4
The asymmetric redundancy of gesture and speech

 | Tufts University
Published online: 26 April 2017
DOI logo with link
https://doi.org/10.1075/gs.7.04der
Abstract
A number of studies from the last decades have demonstrated that iconic gestures are shaped not only by our mental imagery but also, quite strongly, by structural properties of the accompanying speech. These findings are problematic for the central assumption in the Sketch Model (De Ruiter, 2000) about the function of representational gesture. I suggest a seemingly small but fundamental modification to the processing assumptions in the Sketch Model that not only accommodates the discussed empirical findings, but also explains many other well-known gesture phenomena. The new model also generates new and testable predictions regarding the relationship between gesture and speech.
Article outline
  • Introduction
  • The Sketch Model
  • Effects of language on iconic gesture
    • Gestures are influenced by lexical and syntactic properties of language
    • Gesture and speech go hand in hand
    • Gesture and speech express similar types of information
    • Gestures adapt to the pragmatic aspects of speech
  • Implications for the function of iconic gesture
    • The Interface Model
    • The Sketch Model
  • Discussion
  • Notes
  • References
References (47)
References
Alibali, M., & Goldin-Meadow, S. 1993. “Gesture-speech mismatch and mechanisms of learning: What the hands reveal about a child's state of mind.” Cognitive Psychology 25: 468–523. Google Scholar logo with link to Google Scholar
Arbib, M., Liebal, K., & Pika, S. 2008. “Primate vocalization, gesture, and the evolution of human language.” Current Anthropology 49(6): 1063–1076. Google Scholar logo with link to Google Scholar
Bangerter, A. 2004. “Using pointing and describing to achieve joint focus of attention in dialogue.” Psychological Science 15(6): 415–419. Google Scholar logo with link to Google Scholar
Bangerter, A., De Ruiter, J. P., Mayor, E., & Chevalley, E. 2010. “Gestural and speech patterns in collaborative multimodal referring,” Proceedings of the 4th Conference of the International Society for Gesture Studies. Frankfurt am Oder, Germany.Google Scholar logo with link to Google Scholar
Bierwisch, M., & Schreuder, R. 1992. “From concepts to lexical items.” Cognition 42: 23–60. Google Scholar logo with link to Google Scholar
Butterworth, B., & Hadar, U. 1989. “Gesture, speech, and computational stages: A reply to McNeill.” Psychological Review 96: 168–174. Google Scholar logo with link to Google Scholar
Cassell, J., Bickmore, T., Billinghurst, L., Campbell, K., Chang, H., Vilhjalmsson, H., & Yan, H. 1999. “Embodiment in conversational faces: Rea.” Proceedings of the CHI'99 Conference, 520–527. Google Scholar logo with link to Google Scholar
Church, R. B., & Goldin-Meadow, S. 1986. “The mismatch between gesture and speech as an index of transitional knowledge.” Cognition 23: 43–71. Google Scholar logo with link to Google Scholar
Clark, H. H., & Murphy, G. L. 1983. “Audience design in meaning and reference.” In Language and Comprehension, J. F. Leny & W. Kintsch (eds), 287–299.Google Scholar logo with link to Google Scholar
Corbalis, M. 2003. From Hand to Mouth: The Origins of Language. Princeton: Princeton University Press.Google Scholar logo with link to Google Scholar
De Ruiter, J. P. 1998. Gesture and Speech Production. Doctoral Dissertation. Radboud University Nijmegen.Google Scholar logo with link to Google Scholar
2000. “The production of gesture and speech.” In Language and Gesture, D. McNeill (ed), 284–311. Cambridge, UK: Cambridge University Press. Google Scholar logo with link to Google Scholar
2006. “Can gesticulation help aphasic people speak, or rather, communicate?” Advances in Speech-Language Pathology 8(2): 124–127. Google Scholar logo with link to Google Scholar
2007. “Postcards from the mind: the relationship between thought, imagistic gesture, and speech.” Gesture 7(1):21–38.Google Scholar logo with link to Google Scholar
De Ruiter, J. P., Bangerter, A., & Dings, P. 2012. “The interplay between gesture and speech in the production of referring expression: investigating the tradeoff hypothesis.” Topics in Cognitive Science 4: 232–248. Google Scholar logo with link to Google Scholar
De Ruiter, J. P., & De Beer, C. 2013. “A critical evaluation of models of gesture and speech production for understanding gesture in aphasia.” Aphasiology 27(9): 1015–1030. Google Scholar logo with link to Google Scholar
De Ruiter, J. P., & Wilkins, D. P. 1998. The synchronisation of gesture and speech in Dutch and Arrernte (an Australian Aboriginal language): a cross cultural comparison. Paper presented at the Conférence Oralité et Gestualité, Besançon, France.
Enfield, N. J., Kita, S., & De Ruiter, J. P. 2007. “Primary and secondary pragmatic functions of pointing gestures.” Journal of Pragmatics 39(10): 1722–1741. Google Scholar logo with link to Google Scholar
Engle, R. A. 1998. Not channels but composite signals: Speech, gesture, diagrams, and object demonstrations in explanations of mechanical devices. Paper presented at the Twentieth Annual Conference of the Cognitive Science Society, Madison, Wisconsin.
Goldin-Meadow, S., Alibali, M., & Church, R. B. 1993. “Transitions in concept acquisition: Using the hand to read the mind.” Psychological Review 100: 279–297. Google Scholar logo with link to Google Scholar
Gullberg, M. 2011. “Multilingual multimodality: Communicative difficulties and their solutions in second language use.” In Embodied Interaction, J. Streeck, C. Goodwin, & C. LeBaron (eds), Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Hamming, R. W. 1950. “Error correction and error detection codes.” Bell Systems Technical Journal 29(2): 147–160. Google Scholar logo with link to Google Scholar
Kendon, A. 1972. “Some relationships between body motion and speech.” In Studies in Dyadic Communication, A. W. Sigman & B. Pope (eds), New York: Pergamon Press. Google Scholar logo with link to Google Scholar
1975. “Gesticulation, speech, and the gesture theory of language origins.” Sign Language Studies 9(1): 349–373. Google Scholar logo with link to Google Scholar
1980. “Gesticulation and speech: two aspects of the process of an utterance.“ In The Relationship of Verbal and Nonverbal Communication, M. R. Key (ed), 207–227. The Hague: Mouton.Google Scholar logo with link to Google Scholar
2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kita, S. 2003. Pointing: Where Language, Culture, and Cognition Meet: Psychology Press.Google Scholar logo with link to Google Scholar
Kita, S., & Özyürek, A. 2003. “What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking.” Journal of Memory and Language 48: 16–32. Google Scholar logo with link to Google Scholar
Kopp, S., Tepper, P., & Cassell, J. 2004. “Towards integrated microplanning of language and iconic gesture for multimodal output.” Proceedings of 6th International Conference on Multimodal Interfaces. Google Scholar logo with link to Google Scholar
Krauss, R. M., Morrel-Samuels, P., & Colasante, C. 1991. “Do conversational hand gestures communicate?” Journal of Personality & Social Psychology 61(5): 743–754. Google Scholar logo with link to Google Scholar
Levelt, W. J. M. 1989. Speaking: From Intention to Articulation. Cambridge, Massachusetts: The MIT Press.Google Scholar logo with link to Google Scholar
Levelt, W. J. M., Roelofs, A., & Meyer, A. 1999. “A theory of lexical access in speech production.” Behavioral and Brain Sciences 22: 1–75. Google Scholar logo with link to Google Scholar
Mayberry, R. 2000. “Gesture production during stuttered speech: insights into the nature of gesture-speech integration.” In Language and Gesture, D. McNeill (ed), 199–214. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
McClave, E. Z. 1994. “Gestureal beats: The rhythm hypothesis.” Journal of Psycholinguistic Research 23(1): 45–66. Google Scholar logo with link to Google Scholar
McNeill, D. 1985. “So you think gestures are nonverbal?” Psychological Review 92: 350–371. Google Scholar logo with link to Google Scholar
1992. Hand and Mind. Chicago, London: The Chicago University Press.Google Scholar logo with link to Google Scholar
McNeill, D., & Duncan, S. 2000. “Growth points in thinking-for-speaking.” In Language and Gesture, D. McNeill (ed), 141–161. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Morrel-Samuels, P., & Krauss, R. M. 1992. “Word familiarity predicts temporal asynchrony of hand gestures and speech.” Journal of Experimental Psychology: Learning, Memory, and Cognition 18(3): 615–622.Google Scholar logo with link to Google Scholar
Roelofs, A. 1992. “A spreading activation theory of lemma retrieval in speaking.” Cognition 42: 107–142. Google Scholar logo with link to Google Scholar
Rose, M. 2006. “The utility of arm and hand gestures in the treatment of aphasia.” International Journal of Speech-Language Pathology 8(2): 92–109. Google Scholar logo with link to Google Scholar
2013. “Releasing the constraints on aphasia therapy: The positive impact of gesture and multimodality treatments.” American Journal of Speech-Language Pathology 22: 227–239. Google Scholar logo with link to Google Scholar
Sacks, H., & Schegloff, E. A. 1979. “Two preferences in the organization of reference to persons in conversation and their interaction.” In Everyday Language: Studies in Ethnomethodology, G. Psathas (ed), 15–21. New York: Irvington.Google Scholar logo with link to Google Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G. 1974. “A simplest systematics for the organization of turn-taking for conversation.” Language 50: 696–735. Google Scholar logo with link to Google Scholar
Schegloff, E. A. 1984. “On some gesture relation to speech.” In Structures of Social Action: Studies in Conversational Analysis, J. M. Atkinson & J. Heritage (eds), Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Seyfeddinipur, M. 2006. Disfluency: Interrupting speech and gesture. (PhD Thesis), Radboud University Nijmegen.Google Scholar logo with link to Google Scholar
So, W. C., Kita, S., & Goldin-Meadow, S. 2009. “Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand.” Cognitive Science 33: 115–125. Google Scholar logo with link to Google Scholar
Van der Sluis, I., & Krahmer, E. 2007. “Generating multimodal references.” Discourse Processes 44(3): 145–174. Google Scholar logo with link to Google Scholar
Cited by (14)

Cited by 14 other publications

Gawne, Lauren
2025. Gesture, DOI logo
van Maastricht, Lieke & Núria Esteve‐Gibert
2025. Head Gestures Do Not Serve as Precursors of Prosodic Focus Marking in the Second Language as They Do in the First Language. Language Learning DOI logo
de Kleine, Naomi, Miranda L. Rose, Michael Weinborn, Robert Knox & Nicolas Fay
2024. Does Gesture Improve the Communication Success of People with Aphasia?: A Systematic Review. Aphasiology 38:3 ► pp. 462 ff. DOI logo
Wilson, Jack
2024. Pragmatics, Utterance Meaning, and Representational Gesture, DOI logo
Kita, Sotaro & Karen Emmorey
2023. Gesture links language and cognition for spoken and signed languages. Nature Reviews Psychology 2:7 ► pp. 407 ff. DOI logo
Cravotta, Alice, Pilar Prieto & M. Grazia Busà
2021. Exploring the effects of restraining the use of gestures on narrative speech. Speech Communication 135 ► pp. 25 ff. DOI logo
Pouw, Wim, Lisette de Jonge‐Hoekstra, Steven J. Harrison, Alexandra Paxton & James A. Dixon
2021. Gesture–speech physics in fluent speech and rhythmic upper limb movements. Annals of the New York Academy of Sciences 1491:1 ► pp. 89 ff. DOI logo
Stark, Brielle C., Sharice Clough & Melissa Duff
2021. Suggestions for Improving the Investigation of Gesture in Aphasia. Journal of Speech, Language, and Hearing Research 64:10 ► pp. 4004 ff. DOI logo
Clough, Sharice & Melissa C. Duff
2020. The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders. Frontiers in Human Neuroscience 14 DOI logo
de Beer, Carola, Katharina Hogrefe, Martina Hielscher‐Fastabend & Jan P. de Ruiter
2020. Evaluating Models of Gesture and Speech Production for People With Aphasia. Cognitive Science 44:9 DOI logo
Marentette, Paula, Reyhan Furman, Marcus E. Suvanto & Elena Nicoladis
2020. Pantomime (Not Silent Gesture) in Multimodal Communication: Evidence From Children’s Narratives. Frontiers in Psychology 11 DOI logo
Bryła-Cruz, Agnieszka
2019. Communicative responsibility in non-native speech. Overcoming foreign accent in English in religious discourse. Crossroads. A Journal of English Studies :26(3) ► pp. 4 ff. DOI logo
Cravotta, Alice, M. Grazia Busà & Pilar Prieto
2019. Effects of Encouraging the Use of Gestures on Speech. Journal of Speech, Language, and Hearing Research 62:9 ► pp. 3204 ff. DOI logo
de Beer, Carola, Jan P. de Ruiter, Martina Hielscher-Fastabend & Katharina Hogrefe
2019. The Production of Gesture and Speech by People With Aphasia: Influence of Communicative Constraints. Journal of Speech, Language, and Hearing Research 62:12 ► pp. 4417 ff. DOI logo

This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue