We present our work on the temporal integration of hierarchies of communicative actions: kinesic, prosodic and discursive. We use the device of the ‘catchment’ as the locus around which this integration proceeds. We present a detailed case study of a gesture and speech elicitation experiment in which a subject describes her living space to an interlocutor. First, we process the video data to obtain the motion traces of both of the subject’s hands using the vector coherence mapping algorithm. We code the gestures to identify the catchments. We recover discourse purposes utilizing a system of guided questions. Finally, we define prosody in terms of the ToBI system. The results of these analyses are compared against the computed motion traces to identify the cues accessible in the gestural and audio data that correlate well with the psycholinguistic analyses. The results show that motion, prosody and discourse structure are integrated at each moment of speaking.
The electronic edition of this article includes audio-visual data.
2025. IDIOGESTS: GESTURAL IDIOLECTS REVEAL VARIATION IN DISCURSIVE FOCUS. Lege artis. Language yesterday, today, tomorrow► pp. 54 ff.
Moneglia, Massimo & Giorgina Cantalini
2025. Prosodic synchrony and the semantic affiliation of gestures in the multimodal utterance. Journal of Speech Sciences 14 ► pp. e025011 ff.
Calbris, Geneviève & Mary M. Copple
2024. The Gestural Sign: A Concrete and Reasoned Analysis of Co-Speech Gesture. In The Cambridge Handbook of Gesture Studies, ► pp. 217 ff.
Laparle, Schuyler, Gaëlle Ferré & Merel C. J. Scholman
2024. More Than One Gesture but Less Than Two? Inter-stroke Dependencies in Form and Meaning. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management [Lecture Notes in Computer Science, 14711], ► pp. 245 ff.
2024. Facial expressions in different communication settings: A case of whispering and speaking with a face mask in Farsi. Language and Cognition 16:4 ► pp. 1639 ff.
Mortimer, Eduardo Fleury & Renata Reis Pereira
2024. Recurrent gestures in organic chemistry in tertiary education: creating emblems through material and embodied actions. Research in Science & Technological Education 42:1 ► pp. 54 ff.
Ardita, Gabriella
2023. Le labializzazioni su focus informativi e contrastivi nella LIS Una marca pragmatica in comparazione con i gesti coverbali. In Segni, gesti e parole Studi sulla lingua dei segni italiana e su fenomeni di contatto intermodale,
Dachkovsky, Svetlana, Rose Stamp & Wendy Sandler
2023. Mapping the body to the discourse hierarchy in sign language emergence. Language and Cognition 15:1 ► pp. 53 ff.
Donovan, Andrea Marquardt, Sarah A. Brown & Martha W. Alibali
2023. The grammaticalization and dissolution of High Extended Intonation: an inalienable possession paradigm in Roon. STUF - Language Typology and Universals 76:3 ► pp. 403 ff.
Farsani, Danyal, Troels Lange & Tamsin Meaney
2022. Gestures, systemic functional linguistics and mathematics education. Mind, Culture, and Activity 29:1 ► pp. 75 ff.
Gardner, Samuel & Nicholas J. Shea
2022. Gestural Perspectives on Popular-Music Performance. Music Theory Online 28:3
2022. Gestures in Mathematics Thinking and Learning. In Handbook of Cognitive Mathematics, ► pp. 685 ff.
Rodríguez, Lydia
2022. Dickens in Chol. Language and Cognition 14:2 ► pp. 303 ff.
Laparle, Schuyler
2021. Tracking Discourse Topics in Co-speech Gesture. In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Human Body, Motion and Behavior [Lecture Notes in Computer Science, 12777], ► pp. 233 ff.
Pouw, Wim & James A. Dixon
2020. Gesture Networks: Introducing Dynamic Time Warping and Network Analysis for the Kinematic Study of Gesture Ensembles. Discourse Processes 57:4 ► pp. 301 ff.
Alibali, Martha W., Mitchell J. Nathan, Rebecca Boncoddo & Elizabeth Pier
2019. Managing common ground in the classroom: teachers use gestures to support students’ contributions to classroom discourse. ZDM 51:2 ► pp. 347 ff.
Krause, Christina M. & Alexander Salle
2018. On the Role of Gestures for the Descriptive Analysis of ‘Grundvorstellungen’: A Case of Linear Functions. In Signs of Signification [ICME-13 Monographs, ], ► pp. 293 ff.
Lempert, Michael
2018. On the Pragmatic Poetry of Pose: Gesture, Parallelism, Politics. Signs and Society 6:1 ► pp. 120 ff.
Sabena, Cristina
2018. Exploring the Contribution of Gestures to Mathematical Argumentation Processes from a Semiotic Perspective. In Invited Lectures from the 13th International Congress on Mathematical Education [ICME-13 Monographs, ], ► pp. 541 ff.
Krivokapić, Jelena, Mark K. Tiede & Martha E. Tyrone
2017. A Kinematic Study of Prosodic Structure in Articulatory and Manual Gestures: Results from a Novel Method of Data Collection. Laboratory Phonology 8:1
Madeo, Renata C. B., Clodoaldo A. M. Lima & Sarajane M. Peres
2017. Studies in automated hand gesture analysis: an overview of functional types and gesture phases. Language Resources and Evaluation 51:2 ► pp. 547 ff.
Sambre, Paul & Kurt Feyaerts
2017. Embodied musical meaning-making and multimodal viewpoints in a trumpet master class. Journal of Pragmatics 122 ► pp. 10 ff.
Cohn, Neil
2016. A multimodal parallel architecture: A cognitive framework for multimodal interactions. Cognition 146 ► pp. 304 ff.
Arzarello, Ferdinando, Ornella Robutti & Mike Thomas
2015. Growth point and gestures: looking inside mathematical meanings. Educational Studies in Mathematics 90:1 ► pp. 19 ff.
McNeill, David
2015. Gesture in Linguistics. In International Encyclopedia of the Social & Behavioral Sciences, ► pp. 109 ff.
Mcneill, David, Elena T. Levy & Susan D. Duncan
2015. Gesture in Discourse. In The Handbook of Discourse Analysis, ► pp. 262 ff.
Radford, Luis & Cristina Sabena
2015. The Question of Method in a Vygotskian Semiotic Approach. In Approaches to Qualitative Research in Mathematics Education [Advances in Mathematics Education, ], ► pp. 157 ff.
Alibali, Martha W., Mitchell J. Nathan, Matthew S. Wolfgram, R. Breckinridge Church, Steven A. Jacobs, Chelsea Johnson Martinez & Eric J. Knuth
2014. How Teachers Link Ideas in Mathematics Instruction Using Speech and Gesture: A Corpus Analysis. Cognition and Instruction 32:1 ► pp. 65 ff.
Arzarello, Ferdinando & Cristina Sabena
2014. Introduction to the Approach of Action, Production, and Communication (APC). In Networking of Theories as a Research Practice in Mathematics Education [Advances in Mathematics Education, ], ► pp. 31 ff.
Arzarello, Ferdinando & Cristina Sabena
2014. Analytic-Structural Functions of Gestures in Mathematical Argumentation Processes. In Emerging Perspectives on Gesture and Embodiment in Mathematics, ► pp. 75 ff.
2014. The Epistemic Role of Gestures: A Case Study on Networking of APC and AiC. In Networking of Theories as a Research Practice in Mathematics Education [Advances in Mathematics Education, ], ► pp. 127 ff.
Krivokapić, Jelena
2014. Gestural coordination at prosodic boundaries and its role for prosodic structure and speech planning processes. Philosophical Transactions of the Royal Society B: Biological Sciences 369:1658 ► pp. 20130397 ff.
Yasui, Eiko
2013. Collaborative idea construction: Repetition of gestures and talk in joint brainstorming. Journal of Pragmatics 46:1 ► pp. 157 ff.
Kröger, Bernd J., Peter Birkholz, Jim Kannampuzha, Emily Kaufmann & Irene Mittelberg
2011. Movements and Holds in Fluent Sentence Production of American Sign Language: The Action-Based Approach. Cognitive Computation 3:3 ► pp. 449 ff.
Mendoza-Denton, Norma & Stefanie Jannedy
2011. Semiotic Layering through Gesture and Intonation. Journal of English Linguistics 39:3 ► pp. 265 ff.
Flecha-García, María L.
2010. Eyebrow raises in dialogue and their relation to discourse structure, utterance function and pitch accents in English. Speech Communication 52:6 ► pp. 542 ff.
Mary-Annick Morel
2010. Déflexivité et décondensation dans le dialogue oral en français : marqueurs grammaticaux, intonation, regard et geste. Langages n° 178:2 ► pp. 115 ff.
Müller, Cornelia & Tag, Susanne
2010. The Dynamics of Metaphor: Foregrounding and Activating Metaphoricity in Conversational Interaction. Cognitive Semiotics 10:6 ► pp. 85 ff.
Colletta, Jean-Marc, Ramona N. Kunene, Aurélie Venouil, Virginie Kaufmann & Jean-Pascal Simon
2009. Multi-track Annotation of Child Language and Gestures. In Multimodal Corpora [Lecture Notes in Computer Science, 5509], ► pp. 54 ff.
Dick, Anthony Steven, Susan Goldin‐Meadow, Uri Hasson, Jeremy I. Skipper & Steven L. Small
2009. Co‐speech gestures influence neural activity in brain regions associated with processing semantic information. Human Brain Mapping 30:11 ► pp. 3509 ff.
Kataoka, Kuniyoshi
2009. A multi-modal ethnopoetic analysis (Part 1): Text, gesture, and environment in Japanese spatial narrative. Language & Communication 29:4 ► pp. 287 ff.
Kataoka, Kuniyoshi
2010. A multi-modal ethnopoetic analysis (Part 2): Catchment, prosody, and frames of reference in Japanese spatial narrative. Language & Communication 30:2 ► pp. 69 ff.
Levy, Elena T.
2009. The Mediation of Coherent Discourse by Kinesthetic Reenactment: A Case Study of an Autistic Adolescent, Part II. Imagination, Cognition and Personality 29:1 ► pp. 41 ff.
Mittelberg, Irene & Linda R. Waugh
2009. Chapter 14. Metonymy first, metaphor second: A cognitivesemiotic approach to multimodal figures of thought in co-speech gesture. In Multimodal Metaphor, ► pp. 329 ff.
2009. RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication, ► pp. 1022 ff.
Yildirim, Serdar & Shrikanth Narayanan
2009. Automatic Detection of Disfluency Boundaries in Spontaneous Speech of Children Using Audio–Visual Information. IEEE Transactions on Audio, Speech, and Language Processing 17:1 ► pp. 2 ff.
Becvar, Amaya, James Hollan & Edwin Hutchins
2008. Representational Gestures as Cognitive Artifacts for Developing Theories in a Scientific Laboratory. In Resources, Co-Evolution and Artifacts [Computer Supported Cooperative Work, ], ► pp. 117 ff.
Campbell, Nick
2008. Expressive/Affective Speech Synthesis. In Springer Handbook of Speech Processing, ► pp. 505 ff.
Campbell, Nick
2008. Individual Traits of Speaking Style and Speech Rhythm in a Spoken Discourse. In Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction [Lecture Notes in Computer Science, 5042], ► pp. 107 ff.
Cienki, Alan & Cornelia Müller
2008. Metaphor, Gesture, and Thought. In The Cambridge Handbook of Metaphor and Thought, ► pp. 483 ff.
2008. Introduction to the special issue on multimodal corpora for modeling human multimodal behavior. Language Resources and Evaluation 42:2 ► pp. 253 ff.
Mittelberg, Irene
2007. Gesture & Thought. David McNeill, Chicago: Chicago University Press, 2005, 340 pages, $38.00 (hardback). ISBN: 0-226-51462-5.. Metaphor and Symbol 22:3 ► pp. 281 ff.
Nuessel, Frank
2007. Gesture and language. Semiotica 2007:166
Feyereisen, Pierre
2006. Further investigation on the mnemonic effect of gestures: Their meaning matters. European Journal of Cognitive Psychology 18:2 ► pp. 185 ff.
Bryll, R., R.T. Rose & F. Quek
2005. Agent-Based Gesture Tracking. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 35:6 ► pp. 795 ff.
Eisenstein, Jacob & Randall Davis
2004. Proceedings of the 6th international conference on Multimodal interfaces, ► pp. 113 ff.
Eisenstein, Jacob & Randall Davis
2007. ACM SIGGRAPH 2007 courses, ► pp. 15 ff.
Stone, Matthew, Doug DeCarlo, Insuk Oh, Christian Rodriguez, Adrian Stere, Alyssa Lees & Chris Bregler
2004. ACM SIGGRAPH 2004 Papers, ► pp. 506 ff.
Stone, Matthew, Doug DeCarlo, Insuk Oh, Christian Rodriguez, Adrian Stere, Alyssa Lees & Chris Bregler
2004. Speaking with hands. ACM Transactions on Graphics 23:3 ► pp. 506 ff.
2002. Prolegomena of a theory of between-person coordination of speech and gesture. International Journal of Human-Computer Studies 57:4 ► pp. 347 ff.
Quek, Francis, David McNeill, Robert Bryll, Susan Duncan, Xin-Feng Ma, Cemil Kirbas, Karl E. McCullough & Rashid Ansari
2002. Multimodal human discourse. ACM Transactions on Computer-Human Interaction 9:3 ► pp. 171 ff.
Yingen Xiong, F. Quek & D. McNeill
2002. Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, ► pp. 179 ff.
Quek, F.
2001. Proceedings 30th Applied Imagery Pattern Recognition Workshop (AIPR 2001). Analysis and Understanding of Time Varying Imagery, ► pp. 80 ff.
Quek, F., R. Bryll, M. Harper, Lei Chen & L. Ramig
2001. Proceedings 2nd Annual IEEE International Symposium on Bioinformatics and Bioengineering (BIBE 2001), ► pp. 245 ff.
This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.
