Cover not available

Article published In: Functions of Language
Vol. 23:2 (2016) ► pp.149–178

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

The grammatical potential of co-speech gesture

A Functional Discourse Grammar perspective

 | Vrije Universiteit Amsterdam
Published online: 6 October 2016
DOI logo with link
https://doi.org/10.1075/fol.23.2.01kok
In recent years, consensus has been growing that certain co-speech gestures must be incorporated in grammatical theory. Contrary to traditional beliefs, gestural expression has been found to be systematic in various ways, to exhibit some degree of compositional structure and to be to a certain extent language-specific. Most current work on the gesture-grammar interface is, nonetheless, rather programmatic in character or lacks an appropriate theoretical framework. This paper shows that Functional Discourse Grammar (FDG) provides a promising model for understanding the grammar-gesture relationship in a more comprehensive and detailed way than has thus far been achieved. It first evaluates how a FDG perspective on language structure aligns with the ways gestures have been studied. Then it demonstrates that various types of co-speech manual gestures can be classified (perhaps even exhaustively) according to the pragmatic and semantic layers recognized in FDG. On the basis of this survey, a first sketch of a multi-channel (speech-gesture) version of FDG’s architecture is presented. Finally, some challenges are put forward that arise when taking the multimodal grammar enterprise seriously. Altogether, the content of this paper suggests that a multimodal perspective on grammar can be mutually beneficial for functionally oriented linguists and gesture researchers.
References (105)
Alahverdzhieva, Katya. 2013. Alignment of speech and co-speech gesture in a constraint-based grammar. Edinburgh: University of Edinburgh PhD thesis.Google Scholar logo with link to Google Scholar
Allwood, Jens. 2002. Bodily communication dimensions of expression and content. In Inger Karlsson, Björn Granström & David House (eds.), Multimodality in language and speech systems, 7–26. Dordrecht: Kluwer. Google Scholar logo with link to Google Scholar
Andrén, Mats. 2010. Children’s gestures from 18 to 30 months. Lund: Lund University PhD thesis.Google Scholar logo with link to Google Scholar
Anstey, Matthew P. 2008. Functional Discourse Grammar: Multifunctional problems and constructional solutions. Linguistics 46(4). 831–859. Google Scholar logo with link to Google Scholar
Bavelas, Janet B., Nicole Chovil, Douglas A. Lawrie & Allan Wade. 1992. Interactive gestures. Discourse Processes 15(4). 469–489. Google Scholar logo with link to Google Scholar
Beattie, Geoffrey & Heather Shovelton. 1999. Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech. Journal of Language and Social Psychology 18(4). 438–462. Google Scholar logo with link to Google Scholar
Bergmann, Kirsten & Stefan Kopp. 2010. Systematicity and idiosyncrasy in iconic gesture use: Empirical analysis and computational modeling. In Stefan Kopp & Ipke Wachsmuth (eds.), Gesture in embodied communication and human-computer interaction, 182–194. Berlin: Springer. Google Scholar logo with link to Google Scholar
Bressem, Jana. 2012. Repetitions in gesture: Structures, functions, and cognitive aspects. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar logo with link to Google Scholar
Butler, Christopher S. 2008. Cognitive adequacy in structural-functional theories of language. Language Sciences 30(1). 1–30. Google Scholar logo with link to Google Scholar
Calbris, Geneviève. 1990. The semiotics of French gestures. Bloomington, IN: Indiana University Press.Google Scholar logo with link to Google Scholar
. 2008. From left to right: Coverbal gestures and their symbolic use of space. In Alan Cienki & Cornelia Müller (eds.), Metaphor and gesture, 27–53. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Chomsky, Noam. 1983. Noam Chomsky’s views on the psychology of language and thought. In Robert W. Rieber (ed.), Dialogues on the psychology of language and thought, 29–63. New York, NY: Plenum.Google Scholar logo with link to Google Scholar
Cienki, Alan. 2005. Image schemas and gesture. In Beate Hampe & Joseph E. Grady (eds.), From perception to meaning: Image schemas in cognitive linguistics, 421–441. Berlin: Mouton. Google Scholar logo with link to Google Scholar
. 2012. Usage events of spoken language and the symbolic units we (may) abstract from them. In Krzysztof Kosecki & Janusz Badio (eds.), Cognitive processes in language, 149–158. Frankfurt am Main: Lang.Google Scholar logo with link to Google Scholar
. 2015. Spoken language usage events. Language and Cognition 7(4). 499–514. Google Scholar logo with link to Google Scholar
Connolly, John H. 2010. Accommodating multimodality in Functional Discourse Grammar. Web Papers in Functional Discourse Grammar 831. 1–18.Google Scholar logo with link to Google Scholar
De Ruiter, Jan-Peter. 2000. The production of gesture and speech. In David McNeill (ed.), Language and gesture, 331–357. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Debras, Camille & Alan Cienki. 2012. Some uses of head tilts and shoulder shrugs during human interaction, and their relation to stancetaking. Proceedings of the International Conference on Social Computing (SocialCom) , 932–937.
Dik, Simon C. 1986. On the notion ‘Functional Explanation’. Belgian Journal of Linguistics 11. 11–52. Google Scholar logo with link to Google Scholar
Efron, David. 1972. Gesture, race and culture. The Hague: Mouton.Google Scholar logo with link to Google Scholar
Ekman, Paul & Wallace V. Friesen. 1969. Nonverbal leakage and clues to deception. Psychiatry 321. 88–106. Google Scholar logo with link to Google Scholar
. 2004. On linear segmentation and combinatorics in co-speech gesture: A symmetry-dominance construction in Lao fish trap descriptions. Semiotica 1491. 57–124.Google Scholar logo with link to Google Scholar
. 2009. The anatomy of meaning: Speech, gesture, and composite utterances. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Everett, Daniel. In preparation. Sculpting language: A review of the David McNeill Gesture Trilogy. Available online at [URL].
Feyereisen, Pierre, Michèle van de Wiele & Fabienne Dubois. 1988. The meaning of gestures: What can be understood without speech? European Bulletin of Cognitive Psychology 81. 3–25.Google Scholar logo with link to Google Scholar
Firbas, Jan. 1964. On defining the theme in functional sentence analysis. Travaux linguistiques de Prague 11. 267–280.Google Scholar logo with link to Google Scholar
Fricke, Ellen. 2008. Grundlagen einer multimodalen Grammatik des Deutschen. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar logo with link to Google Scholar
. 2009. Multimodal attribution: How gestures are syntactically integrated into spoken language. Paper presented at the International workshop on Gesture and Speech in Interaction (GESPIN), Poznań.
. 2012. Grammatik multimodal: wie Wörter und Gesten zusammenwirken. Berlin: De Gruyter. Google Scholar logo with link to Google Scholar
. 2013. Towards a unified grammar of gesture and speech: A multimodal approach. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 733–754.Google Scholar logo with link to Google Scholar
Habets, Boukje, Sotaro Kita, Zeshu Shao, Asli Özyürek & Peter Hagoort. 2011. The role of synchrony and ambiguity in speech–gesture integration during comprehension. Journal of Cognitive Neuroscience 23(8). 1845–1854. Google Scholar logo with link to Google Scholar
Harrison, Simon. 2009. Grammar, gesture, and cognition: The case of negation in English. Bordeaux: University of Bordeaux PhD thesis.Google Scholar logo with link to Google Scholar
Hassemer, Julius, Gina Joue, Klaus Willmes & Irene Mittelberg. 2011. Dimensions and mechanisms of form constitution: Towards a formal description of gestures. Proceedings of the 2nd Conference on Gesture and Speech in Interaction (GESPIN) .
Haviland, John B. 1993. Anchoring, iconicity, and orientation in Guugu Yimithirr pointing gestures. Journal of Linguistic Anthropology 31. 3–45. Google Scholar logo with link to Google Scholar
Hengeveld, Kees & Evelien Keizer. 2011. Non-straightforward communication. Journal of Pragmatics 43(7). 1962–1976. Google Scholar logo with link to Google Scholar
Hengeveld, Kees & J. Lachlan Mackenzie. 2006. Functional discourse grammar. In Keith Brown (ed.), Encyclopaedia of language and linguistics, vol. 41, 668–676. Oxford: Elsevier. Google Scholar logo with link to Google Scholar
. 2008. Functional Discourse Grammar: A typologically-based theory of language structure. Oxford: OUP. Google Scholar logo with link to Google Scholar
. 2014. Grammar and context in Functional Discourse Grammar. Pragmatics 24(2). 203–227. Google Scholar logo with link to Google Scholar
Kendon, Adam. 1972. Some relationships between body motion and speech. In Aaron W. Siegman & Benjamin Pope (eds.), Studies in dyadic communication, 177–210. Elmsford, NY: Pergamon. Google Scholar logo with link to Google Scholar
. 1988. How gestures can become like words. In Fernando Poyatos (ed.), Cross-cultural perspectives in nonverbal communication, 131–141. Toronto: C. J. Hogrefe.Google Scholar logo with link to Google Scholar
. 1992. Some recent work from Italy on quotable gestures (Emblems). Journal of Linguistic Anthropology 21. 92–108. Google Scholar logo with link to Google Scholar
. 1995. Gestures as illocutionary and discourse structure markers in Southern Italian conversation. Journal of Pragmatics 23(3). 247–279. Google Scholar logo with link to Google Scholar
. 2004. Gesture: Visible action as utterance. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Kendon, Adam, Thomas A. Sebeok & Jean Umiker-Sebeok. 1981. Nonverbal communication, interaction, and gesture: Selections from Semiotica. Berlin: de Gruyter. Google Scholar logo with link to Google Scholar
Kendon, Adam & Laura Versante. 2003. Pointing by hand in Neapolitan. In Sotaro Kita (ed.), 109–137. Pointing: Where language, culture, and cognition meet. Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Kita, Sotaro. 2000. How representational gestures help speaking. In David McNeill (ed.), Language and gesture, 162–185. Cambridge: CUP. Google Scholar logo with link to Google Scholar
. 2003. Pointing: Where language, culture, and cognition meet. New York, NY: Erlbaum. Google Scholar logo with link to Google Scholar
. 2009. Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes 24(2). 145–167. Google Scholar logo with link to Google Scholar
Kita, Sotaro & Asli Özyürek. 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48(1). 16–32. Google Scholar logo with link to Google Scholar
Kress, Gunther R. 2000. Multimodality: Challenges to thinking about language. Tesol Quarterly 34(2). 337–340. Google Scholar logo with link to Google Scholar
Kress, Gunther R. & Theo van Leeuwen. 1996. Reading images: The grammar of visual design. London: Routledge.Google Scholar logo with link to Google Scholar
Kroon, Caroline. 2005. The relationship between grammar and discourse: Evidence from the Latin particle quidem . Latina Lingua. 576–590.Google Scholar logo with link to Google Scholar
Ladewig, Silva H. 2011. Putting the cyclic gesture on a cognitive basis. CogniTextes. Revue de l’Association française de linguistique cognitive 61. Available online at [URL].Google Scholar logo with link to Google Scholar
. 2012. Syntactic and semantic integration of gestures into speech: Structural, cognitive, and conceptual aspects. Frankfurt (Oder): European University Viadrina PhD thesis.Google Scholar logo with link to Google Scholar
Ladewig, Silva H. & Jana Bressem. 2013. New insights into the medium hand: Discovering recurrent structures in gestures. Semiotica 1971. 203–231.Google Scholar logo with link to Google Scholar
Langacker, Ronald W. 2001. Discourse in cognitive grammar. Cognitive Linguistics 12(2). 143–188. Google Scholar logo with link to Google Scholar
. 2008. Cognitive grammar: A basic introduction. Oxford: OUP. Google Scholar logo with link to Google Scholar
Levelt, Wim J.M. 1989. Speaking: From intention to articulation. Cambridge, MA: MIT Press.Google Scholar logo with link to Google Scholar
Levinson, Stephen C. 2003. Space in language and cognition: Explorations in cognitive diversity. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Levy, Elena T. & David McNeill. 1992. Speech, gesture, and discourse. Discourse Processes 15(3). 277–301. Google Scholar logo with link to Google Scholar
Lücking, Andy, Kirsten Bergmann, Hahn Florian, Stefan Kopp & Hannes Rieser. 2010. The Bielefeld speech and gesture alignment corpus (SaGA). Proceedings of the LREC 2010 workshop on Multimodal corpora–advances in capturing, coding and analyzing multimodality.
Mackenzie, J. Lachlan. 2010. More tiles on the roof: Further thoughts on incremental language production. In Kasper Boye & Elisabeth Engberg-Pedersen (eds.), Language usage and language structure, 263–294. Berlin: Mouton.Google Scholar logo with link to Google Scholar
McNeill, David. 1992. Hand and mind. What the hands reveal about thought. Chicago: University of Chicago Press.Google Scholar logo with link to Google Scholar
. 2000. Language and gesture. Cambridge: CUP. Google Scholar logo with link to Google Scholar
. 2003. Aspects of aspect. Gesture 3(1). 1–17. Google Scholar logo with link to Google Scholar
. 2005. Gesture and thought. Chicago, IL: University of Chicago Press. Google Scholar logo with link to Google Scholar
McNeill, David & Susan D. Duncan. 2000. Growth points in thinking-for-speaking. In David McNeill (ed.), 162–185. Google Scholar logo with link to Google Scholar
McNeill, David & Elena Levy. 1982. Conceptual representations in language activity and gesture. In Robert J. Jarvella & Wolfgang Klein (eds.), Speech, place and action: Studies in deixis and related topics. New York, NY: Wiley.Google Scholar logo with link to Google Scholar
Mittelberg, Irene & Tanja Mortelmans. 2013. Grammaticalized manual actions of giving and holding: From ditransitive to existential constructions in speech and gesture. Paper presented at the International Conference of Cognitive Linguistics (ICLC12) , Edmonton, Canada.
Müller, Cornelia. 1998. Iconicity and gesture. In Serge Santi (ed.), Oralité et Gestualité: Communication Multimodale, Interaction, 321–328. Montréal: L’Harmattan.Google Scholar logo with link to Google Scholar
. 2004. Forms and uses of the Palm Up Open Hand: A case of a gesture family. In Cornelia Müller & Roland Posner (eds.), The semantics and pragmatics of everyday gestures, 234–256. Berlin: Weidler Verlag.Google Scholar logo with link to Google Scholar
. 2013. Gestures as a medium of expression: The linguistic potential of gestures. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 202–217.Google Scholar logo with link to Google Scholar
Müller, Cornelia, Jana Bressem & Silva Ladewig. 2013a. Towards a grammar of gestures: A form-based view. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 707–732.Google Scholar logo with link to Google Scholar
Müller, Cornelia, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.). 2013b. Body – Language – Communication: An international handbook on multimodality in human interaction, vol. 11. Berlin: Mouton.Google Scholar logo with link to Google Scholar
Müller, Cornelia, Silva H. Ladewig & Jana Bressem. 2013c. Gestures and speech from a linguistic perspective: A new field and its history. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 55–81.Google Scholar logo with link to Google Scholar
Muntigl, Peter. 2004. Modelling multiple semiotic systems: The case of gesture and speech. In Eija Ventola, Cassily Charles & Martin Kaltenbacher (eds.), Perspectives on multimodality, 31–50. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Okrent, Arika. 2002. A modality-free notion of gesture and how it can help us with the morpheme vs. gesture question in sign language linguistics (or at least give us some criteria to work with). In Richard P. Meier, Kearsy Cormier & David Quinto-Pozos (eds.), Modality and structure in signed and spoken languages, 175–198. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Özyürek, Asli, Sotaro Kita, Shanley E.M. Allen, Reyhan Furman & Amanda Brown. 2005. How does linguistic framing of events influence co-speech gestures? Insights from crosslinguistic variations and similarities. Gesture 5(1–2). 219–240. Google Scholar logo with link to Google Scholar
Parrill, Fey. 2010. Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 251. 650–668. Google Scholar logo with link to Google Scholar
Parrill, Fay, Benjamin K. Bergen & Patricia V. Lichtenstein. 2013. Grammatical aspect, gesture, and conceptualization: Using co-speech gesture to reveal event representations. Cognitive Linguistics 24(1). 135–158. Google Scholar logo with link to Google Scholar
Payrató, Lluís. 1993. A pragmatic view on autonomous gestures: A first repertoire of Catalan emblems. Journal of Pragmatics 20(3). 193–216. Google Scholar logo with link to Google Scholar
Sadeghipour, Amir & Stefan Kopp. 2014. Learning a motor grammar of gesture. Proceedings of the Annual Conference of the Cognitive Science Society , Quebec City, Canada.
Sandoval, Richard. 2014. Gestures in native Northern America: Bimodal talk in Arapaho. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), 1215–1226.Google Scholar logo with link to Google Scholar
Schoonjans, Steven. 2014. Is gesture subject to grammaticalization? Papers of the Linguistic Society of Belgium 81. Available online at [URL]
Sherzer, Joel. 1973. Verbal and nonverbal deixis: The pointed lip gesture among the San Blas Cuna. Language in Society 21. 117–131. Google Scholar logo with link to Google Scholar
Slama-Cazacu, Tatiana. 1976. Nonverbal components in message sequence: ‘Mixed syntax’. In William C. McCormack & Stephen A. Wurm (eds.), Language and man: Anthropological issues, 217–227. The Hague: Mouton.Google Scholar logo with link to Google Scholar
Sparhawk, Carol M. 1978. Contrastive-identificational features of Persian gesture. Semiotica 24(1–2). 49–86. Google Scholar logo with link to Google Scholar
Stokoe, William C. 1960/2005. Sign language structure: An outline of the visual communication systems of the American deaf. Journal of Deaf Studies and Deaf Education 10(1). 3–37. Google Scholar logo with link to Google Scholar
Streeck, Jürgen. 1993. Gesture as communication: Its coordination with gaze and speech. Communications Monographs 60(4). 275–299. Google Scholar logo with link to Google Scholar
. 2002. Grammars, words, and embodied meanings: On the uses and evolution of so and like . Journal of Communication 52(3). 581–596. Google Scholar logo with link to Google Scholar
Sweetser, Eve. 1998. Regular metaphoricity in gesture: Bodily-based models of speech interaction. Actes du 16e Congrès International des Linguistes .
Talmy, Leonard. 1985. Lexicalization patterns: Semantic structure in lexical forms. In Timothy Shopen (ed.), Language typology and syntactic description Vol. 3: Grammatical categories and the lexicon, 57–149. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Turner, Mark & Francis Steen. 2012. Multimodal Construction Grammar. In Michael Borkent, Barbara Dancygier & Jennifer A.J. Hinnell (eds.), Language and the creative mind, 255–274. Stanford, CA: CSLI Publications.Google Scholar logo with link to Google Scholar
Webb, Rebecca A. 1996. Linguistic features of metaphoric gestures. New York, NY: University of Rochester PhD thesis.Google Scholar logo with link to Google Scholar
Wilcox, Sherman. 2004. Cognitive iconicity: Conceptual spaces, meaning, and gesture in signed language. Cognitive Linguistics 15(2). 119–148. Google Scholar logo with link to Google Scholar
Wilcox, Sherman & André Nogueira Xavier. 2013. A framework for unifying spoken language, signed language, and gesture. Revista Todas as Letras 15(1). 88–110.Google Scholar logo with link to Google Scholar
Wilkins, David. 2003. Why pointing with the index finger is not a universal (in sociocultural and semiotic terms). In Sotaro Kita (ed.), 171–215.Google Scholar logo with link to Google Scholar
Zima, Elisabeth. 2014. English multimodal motion constructions: A Construction Grammar perspective. Papers of the Linguistic Society of Belgium 81. Available online at [URL]
Cited by (7)

Cited by seven other publications

Bateman, John A.
2025. Multimodality writes back. Language, Context and Text. The Social Semiotics Forum 7:1 ► pp. 1 ff. DOI logo
Cornish, Francis
2022. Text, discourse, context: A meta-trilogy for discourse analysis. Journal of Pragmatics 199 ► pp. 91 ff. DOI logo
Laparle, Schuyler
2022. The Interaction Space. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Anthropometry, Human Behavior, and Communication [Lecture Notes in Computer Science, 13319], ► pp. 243 ff. DOI logo
Martin, J. R. & Michele Zappavigna
2019. Embodied meaning: a systemic functional perspective on paralanguage. Functional Linguistics 6:1 DOI logo
Lepic, Ryan & Corrine Occhino
2018. A Construction Morphology Approach to Sign Language Analysis. In The Construction of Words [Studies in Morphology, 4], ► pp. 141 ff. DOI logo
Kok, Kasper I.
2017. Functional and temporal relations between spoken and gestured components of language. International Journal of Corpus Linguistics 22:1 ► pp. 1 ff. DOI logo
Mittelberg, Irene
2017. Multimodal existential constructions in German: Manual actions of giving as experiential substrate for grammatical and gestural patterns. Linguistics Vanguard 3:s1 DOI logo

This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue