Cover not available

Article published In: Gesture
Vol. 15:1 (2016) ► pp.37–59

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Mapping out the multifunctionality of speakers’ gestures

 | Vrije Universiteit Amsterdam
 | Bielefeld University
 | Moscow State Linguistic University
Published online: 19 May 2016
DOI logo with link
https://doi.org/10.1075/gest.15.1.02kok
Although it is widely acknowledged that gestures are complex functional elements of human communication, many current functional classification systems are rather rigid, implicitly assuming gestures to perform only one function at any given time. In this paper, we present a theoretical view on the inherent multifunctionality of speakers’ gestures, inspired by frameworks in structural-functional linguistics (Halliday’s Systemic Functional Grammar and Hengeveld & Mackenzie’s Functional Discourse Grammar). Building upon this view, we report on a large-scale internet-based gesture perception study, designed in a way open to the potential for complex multifunctionality of gestural expression. The results provide quantitative support for the view that speakers’ gestures typically contribute to multiple semantic and meta-communicative functions of the ongoing discourse simultaneously. Furthermore, we identify clusters of functions that tend to be combined, as well as correlations between pairs of them. As a whole, this paper achieves some degree of convergence between ecological and experimental views on gesture functionality.
References (42)
Bavelas, Janet Beavin, Nicole Chovil, Douglas A. Lawrie, & Allan Wade (1992). Interactive gestures. Discourse Processes, 15 (4), 469–489. Google Scholar logo with link to Google Scholar
Beattie, Geoffrey & Heather Shovelton (1999). Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology, 18 (4), 438–462. Google Scholar logo with link to Google Scholar
Bergmann, Kirsten & Stefan Kopp (2006). Verbal or visual? How information is distributed across speech and gesture in spatial dialog. In David Schlangen & Raquel Fernández (Eds.), Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue (SemDial-10) (pp. 90–97). Potsdam, Germany: Universitätsverlag.Google Scholar logo with link to Google Scholar
(2009). Increasing expressiveness for virtual agents – Autonomous generation of speech and gesture for spatial description tasks. In Keith S. Decker, Jaime S. Sichman, Carles Sierra, & Cristiano Castelfranchi (Eds.), Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2009) (pp. 361–368). Ann Arbor, MI: IFAAMAS.Google Scholar logo with link to Google Scholar
Bergmann, Kirsten, Volkan Aksu, & Stefan Kopp (2011). The relation of speech and gestures: Temporal synchrony follows semantic synchrony. In Proceedings of the 2nd Workshop on Gesture and Speech in Interaction (GeSpIn 2011) . Bielefeld, Germany.
Bressem, Jana, Silva H. Ladewig, & Cornelia Müller (2013). Linguistic Annotation System for Gestures (LASG). In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Teßendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 1098–1125). Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Bühler, Karl (1990/1934). Theory of language: The representational function of language, trans. Donald Fraser Goodwin. Amsterdam: John Benjamins (Sprachtheorie. Die Darstellungsfunktion der Sprache. Jena: Gustav Fischer). Google Scholar logo with link to Google Scholar
Clark, Herbert H. & Jean E. Fox Tree (2002). Using uh and um in spontaneous speaking. Cognition, 84 (1), 73–111. Google Scholar logo with link to Google Scholar
Connolly, John H. (2010). Accommodating multimodality in Functional Discourse Grammar. Web Papers in Functional Discourse Grammar, 831, 1–18.Google Scholar logo with link to Google Scholar
Denis, Michel (1997). The description of routes: A cognitive approach to the production of spatial discourse. Current Psychology of Cognition, 161, 409–458.Google Scholar logo with link to Google Scholar
Dik, Simon C. (1989). The theory of functional grammar, Part I: The structure of the clause. Dordrecht: Foris Publications.Google Scholar logo with link to Google Scholar
Efron, David (1972). Gesture, race and culture. The Hague: Mouton.Google Scholar logo with link to Google Scholar
Enfield, Nicholas J. (2009). The anatomy of meaning: Speech, gesture, and composite utterances. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Halliday, Michael A.K. (1985). An introduction to functional grammar. London: Edward Arnold.Google Scholar logo with link to Google Scholar
Hengeveld, Kees & J. Lachlan Mackenzie (2008). Functional Discourse Grammar: A typologically-based theory of language structure. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Holler, Judith & Geoffrey Beattie (2002). A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk. Semiotica, 1421, 31–69.Google Scholar logo with link to Google Scholar
Holler, Judith & Katie Wilkin (2009). Communicating common ground: How mutually shared knowledge influences speech and gesture in a narrative task. Language and Cognitive Processes, 24 (2), 267–289. Google Scholar logo with link to Google Scholar
Humboldt, Wilhelm von (1905). Gesammelte Schriften, Bd. 41 [The collected works of Wilhelm von Humboldt, Vol. 4]. Berlin: Behr. Google Scholar logo with link to Google Scholar
Kendon, Adam (1972). Some relationships between body motion and speech. Studies in Dyadic Communication, 71, 177–210. Google Scholar logo with link to Google Scholar
(1980). Gesticulation and speech: Two aspects of the process of utterance. In Mary R. Key (Ed.), The relationship of verbal and nonverbal communication (pp. 207–227). The Hague: Mouton.Google Scholar logo with link to Google Scholar
(1981). Introduction: Current issues in the study of nonverbal communication. In Adam Kendon, Thomas A. Sebeok, & Jean Umiker-Sebeok (Eds.), Nonverbal communication, interaction, and gesture: Selections from Semiotica (pp. 1–53). Berlin: Walter de Gruyter. Google Scholar logo with link to Google Scholar
(1995). Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of pragmatics, 23 (3), 247–279. Google Scholar logo with link to Google Scholar
(2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kok, Kasper (in press). The grammatical potential of co-speech gesture: A Functional Discourse Grammar perspective. Functions of Language.
Lausberg, Hedda & Han Sloetjes (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41 (3), 841–849. Google Scholar logo with link to Google Scholar
Levy, Elena T. & David McNeill (1992). Speech, gesture, and discourse. Discourse Processes, 15 (3), 277–301. Google Scholar logo with link to Google Scholar
Loehr, Daniel P. (2004). Gesture and intonation . PhD thesis, Georgetown University.
Lücking, Andy, Kirsten Bergman, Florian Hahn, Stefan Kopp, & Hannes Rieser (2013). Data-based analysis of speech and gesture: The Bielefeld Speech and Gesture Alignment Corpus (SaGA) and its applications. Journal on Multimodal User Interfaces, 7 (1/2), 5–18. Google Scholar logo with link to Google Scholar
Martinec, Radan (2004). Gestures that co‐occur with speech as a systematic resource: The realization of experiential meanings in indexes. Social Semiotics, 14 (2), 193–213. Google Scholar logo with link to Google Scholar
McNeill, David (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar logo with link to Google Scholar
(2005). Gesture and thought. Chicago: University of Chicago Press. Google Scholar logo with link to Google Scholar
Müller, Cornelia (1998). Redebegleitende Gesten. Kulturgeschichte – Theorie – Sprachvergleich. Berlin: Berlin Verlag A. Spitz.Google Scholar logo with link to Google Scholar
(2013). Gestures as a medium of expression: The linguistic potential of gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Tessendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 202–217). Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Müller, Cornelia, Silva Ladewig, & Jana Bressem (2013). Gestures and speech from a linguistic perspective: A new field and its history. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, & Sedinha Tessendorf (Eds.), Body – language – communication: An international handbook on multimodality in human interaction, Vol. 11 (pp. 55–81). Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Muntigl, Peter (2004). Modelling multiple semiotic systems: The case of gesture and speech. In Eija Ventola, Cassily Charles, & Martin Kaltenbacher (Eds.), Perspectives on multimodality (pp. 31–49). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Parrill, Fey (2010). Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes, 25 (5), 650–668. Google Scholar logo with link to Google Scholar
Sperber, Dan & Deirdre Wilson (1986). Relevance: Communication and cognition. Oxford: Blackwell.Google Scholar logo with link to Google Scholar
Van Valin, Robert D., Jr. (1993). Advances in role and reference grammar. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Wundt, Wilhelm (1973). The language of gestures. The Hague: Mouton. Google Scholar logo with link to Google Scholar
Cited by (22)

Cited by 22 other publications

Bavelas, Janet
2024. Gesturing for the Addressee. In The Cambridge Handbook of Gesture Studies, ► pp. 579 ff. DOI logo
Wilson, Anna, Irina Pavlova, Elinor Payne, Ilya Burenko & Peter Uhrig
2024. World futures through RT’s eyes: multimodal dataset and interdisciplinary methodology. Frontiers in Communication 9 DOI logo
Leonteva, Anna V., Alan Cienki & Olga V. Agafonova
2023. Metaphoric gestures in simultaneous interpreting. Russian Journal of Linguistics 27:4 ► pp. 820 ff. DOI logo
Lopez-Ozieblo, Renia
2023. Observations on the Progression of Gestures with L2 Proficiency: A Call for Further Research. Journal of Psycholinguistic Research 52:2 ► pp. 381 ff. DOI logo
Molodychenko, Evgeni N.
2023. Levels of context: how textual analysis becomes discourse analysis: the case of Internet lifestyle media. Slovo.ru: Baltic accent 14:1 ► pp. 107 ff. DOI logo
Cienki, Alan
2022. The study of gesture in cognitive linguistics: How it could inform and inspire other research in cognitive science. WIREs Cognitive Science 13:6 DOI logo
Cienki, Alan
2024. Variable embodiment of stance-taking and footing in simultaneous interpreting. Frontiers in Psychology 15 DOI logo
Hodge, Gabrielle & Lindsay Ferrara
2022. Iconicity as Multimodal, Polysemiotic, and Plurifunctional. Frontiers in Psychology 13 DOI logo
Laparle, Schuyler
2022. The Interaction Space. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication [Lecture Notes in Computer Science, 13319], ► pp. 243 ff. DOI logo
Laparle, Schuyler
2023. Gradient at-issueness and semiotic complexity in gesture: a response. Theoretical Linguistics 49:3-4 ► pp. 261 ff. DOI logo
Uhrig, Peter
2022. Hand gestures with verbs of throwing: Collostructions, style and metaphor. Yearbook of the German Cognitive Linguistics Association 10:1 ► pp. 99 ff. DOI logo
Jehlička, Jakub & Eva Lehečková
2020. Multimodal Event Construals: The Role of Co-Speech Gestures in English vs. Czech Interactions. Zeitschrift für Anglistik und Amerikanistik 68:4 ► pp. 351 ff. DOI logo
Afonso, Susana & Ana Sofia Afonso
2019. Accessing Science Museum Educators' Discourse Through Multimodal Narratives. In Multimodal Narratives in Research and Teaching Practices [Advances in Educational Technologies and Instructional Design, ], ► pp. 340 ff. DOI logo
Feng, Shichang, Zhiquan Feng & Liujuan Cao
2019. Many-to-One Gesture-to-Command Flexible Mapping Approach for Smart Teaching Interface Interaction. IEEE Access 7 ► pp. 179517 ff. DOI logo
Gawne, Lauren, Chelsea Krajcik, Helene N. Andreassen, Andrea L. Berez-Kroeker & Barbara F. Kelly
2019. Data transparency and citation in the journal Gesture . Gesture 18:1 ► pp. 83 ff. DOI logo
Lederer, Jenny
2019. Gesturing the source domain. Metaphor and the Social World 9:1 ► pp. 32 ff. DOI logo
Martin, J. R. & Michele Zappavigna
2019. Embodied meaning: a systemic functional perspective on paralanguage. Functional Linguistics 6:1 DOI logo
Austin, Elizabeth E. & Naomi Sweller
2018. Gesturing Along the Way: Adults’ and Preschoolers’ Communication of Route Direction Information. Journal of Nonverbal Behavior 42:2 ► pp. 199 ff. DOI logo
Hassemer, Julius & Bodo Winter
2018. Decoding Gestural Iconicity . Cognitive Science 42:8 ► pp. 3034 ff. DOI logo
Kok, Kasper I.
2017. Functional and temporal relations between spoken and gestured components of language. International Journal of Corpus Linguistics 22:1 ► pp. 1 ff. DOI logo
Parrill, Fey & Kashmiri Stec
2017. Gestures of the abstract. Pragmatics & Cognition 24:1 ► pp. 33 ff. DOI logo
Freigang, Farina & Stefan Kopp
2016. This Is What’s Important – Using Speech and Gesture to Create Focus in Multimodal Utterance. In Intelligent Virtual Agents [Lecture Notes in Computer Science, 10011], ► pp. 96 ff. DOI logo

This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue