Cover not available

Article published In: Gesture
Vol. 23:3 (2024) ► pp.217–258

Get fulltext from our e-platform
JBE logo with link
EPUB logo with download linkDownload EPUB

What counts as a relevant gesture in the study of multimodal event expressions?

ORCiD logo with link | Monash University
ORCiD logo with link | The University of Melbourne
ORCiD logo with link | Monash University
ORCiD logo with link | The University of the South Pacific
Open Access logo
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Open Access publication of this article was funded through a Transformative Agreement with Monash University.

Published online: 26 September 2025
DOI logo with link
https://doi.org/10.1075/gest.25002.mar
Abstract
This article explores how methodological decisions about which gestures to include impact the analysis of multimodal event representation. Gestures commonly temporally align with semantically co-expressive speech. However, when we consider not single concepts but whole events the picture gets more complicated, since events are not mapped onto single lexical items but across larger stretches of spoken language. To research gesture-speech integration in multimodal event expressions we must decide which gestures are considered ‘relevant’. Using caused motion events as a case study, we apply different definitions of gesture inclusion relating to temporal and semantic alignment. We investigate the impact of definition choice on the results by exploring corpus data from Australian English and two endangered Oceanic languages, Saliba-Logea and Sudest. The study shows comparable trends across some conditions but also language- and definition-specific differences. This has a bearing on how confidently we can compare studies based on different methodologies and/or languages.
Article outline
  • Introduction
    • Gesture and event expression
  • Directed caused accompanied motion events
  • Methodology
    • Language corpora
    • Speech annotation
    • Gesture annotation
    • Criteria for gesture inclusion
  • Results
    • Representation of gestures across conditions
    • Representation of meaning components across conditions
      • Capturing of meaning components across conditions
      • Capturing of individual meaning components across conditions
      • Semantic makeup of datasets across condition
    • Meaning components per gesture
  • Discussion and Conclusions
  • Acknowledgements
  • Notes
  • Abbreviations
  • References
References (46)
References
Aikhenvald, A. Y. 2000. Classifiers: A typology of noun categorization devices. Oxford University Press. Google Scholar logo with link to Google Scholar
Akhavan, N., Nazbanou, B. N., & Göksun, T. 2015. Motion event expressions in language and gesture: Evidence from Persian. In Noelle, D. C., Dale, R., Warlaumont, A. S., Yoshimi, J., Matlock, T., Jennings, C. D., & Maglio, P. P. (Eds.), Proceedings of the 37th Annual Meeting of the Cognitive Science Society, CogSci 2015, Pasadena, California, USA, July 22–25, 2015.Google Scholar logo with link to Google Scholar
Burke, I., Margetts, A., Plehwe, R., & Poulton, T. Forthcoming. The encoding of directed caused accompanied motion events in English: BRING and TAKE and beyond.
Bergmann, K., Aksu, V., & Kopp, S. 2011. The relation of speech and gestures: Temporal synchrony follows semantic synchrony. In Jarmołowicz-Nowikow, E., Juzcyk, K. M., Malisz, S., & Szczyszek, M. L. (Eds.), Proceedings of the 2nd Workshop on Gesture and Speech in Interaction (GeSpIn 2011).Google Scholar logo with link to Google Scholar
Bohnemeyer, J., Enfield, N. J., Essegbey, J., Ibarretxe-Antuñano, I., Kita, S., Lüpke, F., & Ameka, F. K. 2007. Principles of event segmentation in language: The case of motion events. Language 83(3), 495–532. Google Scholar logo with link to Google Scholar
Butterworth, B., & Hadar, U. 1989. Gesture, speech, and computational stages: A reply to McNeill. Psychological Review 96(1), 168–174. Available at: (last access 8 June 2024).Google Scholar logo with link to Google Scholar
Choi, S., & Lantolf, J. P. 2008. Representation and embodiment of meaning in L2 communication: Motion events in the speech and gesture of advanced L2 Korean and L2 English speakers. Studies in Second Language Acquisition 30(2), 191–224. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Debreslioska, S., & Gullberg, M. 2020. What’s new? Gestures accompany inferable rather than brand-new referents in discourse. Frontiers in Psychology 111, 1935. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Defina, R. 2016. Do serial verb constructions describe single events? A study of co-speech gestures in Avatime. Language 92(4), 890–910. Available at: (last access 8 June 2024).Google Scholar logo with link to Google Scholar
Duncan, S. 2002. Gesture, verb aspect, and the nature of iconic imagery in natural discourse. Gesture 21, 183–206. Available at: (last access 8 June 2024).Google Scholar logo with link to Google Scholar
Foley, W. A. 1986. The Papuan languages of New Guinea. Cambridge University Press.Google Scholar logo with link to Google Scholar
Furman, R. 2012. Caused motion events in Turkish: Verbal and gestural representation in adults and children. [Doctoral dissertation, Radboud University Nijmegen]. Available at: [URL] (last access 26 November 2021).
Gullberg, M. 2011. Language-specific encoding of placement events in gestures. In Bohnemeyer, J., & Pederson, E. (Eds.), Event representation in language and cognition, pp. 166–188. Cambridge University Press. Available at: (last access 26 November 2021)Google Scholar logo with link to Google Scholar
Hellwig, B., Margetts, A., Riesberg, S., & Schippling, S. 2022. Bringing and taking: A cross-linguistic perspective on caused accompanied motion events. In Margetts, A., Riesberg, S., & Hellwig, B. (Eds.), Caused accompanied motion: Bringing and taking events in a cross-linguistic perspective (Typological Studies in Language 134), pp. 1–41. John Benjamins. Google Scholar logo with link to Google Scholar
Hendriks, H., Hickmann, M., & Demagny, A. 2008. How adult English learners of French express caused motion: A comparison with English and French natives. Aile: Acquisition et interaction en langue étrangere 271, 15–41. Available at: [URL] (last access 22 July 2025).
Himmelmann, N. 2006. The challenges of segmenting spoken language. In Gippert, J., Himmelmann, N., & Mosel, U. (Eds.), Essentials of language documentation (Trends in Linguistics. Studies and Monographs 178), pp. 253–274. De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Hostetter, A. B., Alibali, M., & Kita, S. 2007. I see it in my hands’ eye: Representational gestures reflect conceptual demands. Language and Cognitive Processes 221, 313–336. Available at: (last access 8 June 2024).Google Scholar logo with link to Google Scholar
Jackendoff, R. 1983. Semantics and cognition (Current Studies in Linguistics Series 8). MIT Press.Google Scholar logo with link to Google Scholar
Kendon, A. 2004. Gesture: Visible action as utterance. Cambridge University Press. Google Scholar logo with link to Google Scholar
Kita, S. 2009. Cross-cultural variation of speech-accompanying gesture: A review. Language and Cognitive Processes 24(2), 145–167. Available at: (last access 8 June 2024).Google Scholar logo with link to Google Scholar
Kita, S., & Özyürek, A. 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48(1), 16–32. Available at: [URL] (last access July 2025)
Levin, B. 1993. English verb classes and alternations: a preliminary investigation. University of Chicago Press.Google Scholar logo with link to Google Scholar
Levy, E. T., & McNeill, D. 1992. Speech, gesture, and discourse. Discourse Processes 15(3), 277–301. Google Scholar logo with link to Google Scholar
Margetts, A. 2022. Directed caused accompanied motion events in Saliba-Logea. In Margetts, A., Riesberg, S., & Hellwig, B. (Eds.), Caused accompanied motion: Bringing and taking events in a cross-linguistic perspective (Typological Studies in Language 134), pp. 147–186. John Benjamins. Google Scholar logo with link to Google Scholar
Margetts, A., & Austin, P. K. 2007. Three-participant events in the languages of the world: towards a crosslinguistic typology. Linguistics 45(3), 393–451. Available at: [URL] (last access 1 April 2010)
Margetts, A., Haude, K., Himmelmann, N. P., Jung, D., Riesberg, S., Schnell, S., Seifart, F., Sheppard, H., & Wegener, C. 2022. Cross-linguistic patterns in the lexicalisation of BRING and TAKE. Studies in Language. 46(4), 934–993. Available at: (last access 1 September 2022).Google Scholar logo with link to Google Scholar
Margetts, A., Riesberg, S., & Hellwig, B. (Eds.). 2022. Caused accompanied motion: Bringing and taking events in a cross-linguistic perspective (Typological Studies in Language 134). John Benjamins. Google Scholar logo with link to Google Scholar
McNeill, D. 1992. Hand and mind: What gestures reveal about thought. Vol. 271. University of Chicago Press. [URL]
Mithun, M. 1999. The languages of native North America (Cambridge Language Surveys). Cambridge University Press.Google Scholar logo with link to Google Scholar
Narasimhan, B., Kopecka, A., Bowerman, M., Gullberg, M., & Majid, A. 2012. Putting and taking events: A crosslinguistic perspective. In Kopecka, A., & Narasimhan, B. (Eds.), Events of putting and taking: A crosslinguistic perspective, pp. 1–20. John Benjamins. Google Scholar logo with link to Google Scholar
Negueruela, E., Lantolf, J. P., Rehn Jordan, S., & Gelabert, J. 2004. The “private function” of gesture in second language speaking activity: A study of motion verbs and gesturing in English and Spanish. International Journal of Applied Linguistics 14(1), 113–147. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Özyürek, A., Kita, S., Allen, S., Brown, A., Furman, R., & Ishizuka, T. 2008. Development of cross-linguistic variation in speech and gesture: Motion events in English and Turkish. Developmental Psychology 44(4), 1040–1054. Available at: [URL] (last access 24 June 2025).
Özyürek, A., Kita, S., Allen, S., Furman, R., & Brown, A. 2005. How does linguistic framing of events influence co-speech gestures? Insights from crosslinguistic variations and similarities. Gesture 5(1–2), 219–240. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
R Core team. 2022. R: A language and environment for statistical computing. R Foundation for Statistical Computing.Google Scholar logo with link to Google Scholar
de Ruiter, J. P. 2000. The production of gesture and speech. In McNeill, D. (Ed.), Language and gesture, pp. 284–311. 1st edn. Cambridge; New York: Cambridge University Press. Available at: (last access 10 June 2025).Google Scholar logo with link to Google Scholar
Schegloff, E. A. 1985. On some gestures’ relation to talk. In Maxwell Atkinson, J. (Ed.), Structures of social action, pp. 266–296. 1st edn. Cambridge University Press. Available at: (last access 10 June 2025).Google Scholar logo with link to Google Scholar
Sheppard, H. 2020. Verbal morphosyntax and three-participant events in Sudest. [Doctoral dissertation Monash University].
2022. Directed caused accompanied motion events in Sudest, an Oceanic language with classificatory verbs. In Margetts, A., Riesberg, S., & Hellwig, B. (Eds.), Caused accompanied motion: Bringing and taking events in a cross-linguistic perspective (Typological Studies in Language 134), pp. 187–217. John Benjamins. Google Scholar logo with link to Google Scholar
Slobin, D. 1996. From “thought to language” to “thinking for speaking.” In Gumperz, J. J., & Levinson, S. C. (Eds.), Rethinking linguistic relativity (Studies in the Social and Cultural Foundations of Language 17), pp. 70–96. Cambridge University Press.Google Scholar logo with link to Google Scholar
2004. The many ways to search for a frog: Linguistic typology and the expression of motion events. In Strömquist, S., & Verhoeven, L. (Eds.), Relating events in narrative: typological and contextual perspectives, pp. 219–257. Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
Stam, G. 2006. Thinking for speaking about motion: L1 and L2 speech and gesture. IRAL — International Review of Applied Linguistics in Language Teaching 44(2), 145–171. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Talmy, L. 1985. Lexicalization patterns: Semantic structure in lexical forms. In Shopen, T. (Ed.), Language typology and syntactic description, Vol. 31: Grammatical categories and the lexicon, pp. 57–149. Cambridge University Press.Google Scholar logo with link to Google Scholar
ter Bekke, M., Drijvers, L., & Holler, J. 2024. Gestures speed up responses to questions. Language, Cognition and Neuroscience 39(4), 423–430. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. 2006. ELAN: A Professional Framework for Multimodality Research. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). Genoa, Italy: European Language Resources Association (ELRA), Available at: [URL] (last access 10 June 2022).
Yoshioka, K. 2008. Gesture and information structure in first and second language. Gesture 81, 236–255. Available at: (last access 24 June 2025).Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue