Cover not available

Article published In: Interaction and Iconicity in the Evolution of Language:
Edited by Stefan Hartmann, Michael Pleyer, James Winters and Jordan Zlatev
[Interaction Studies 18:3] 2017
► pp. 465–488

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Multimodal-first or pantomime-first?

Communicating events through pantomime with and without vocalization

 | Lund University |
 | Nicolaus Copernicus University |
 | Nicolaus Copernicus University |
 | Lund University |
Published online: 8 December 2017
DOI logo with link
https://doi.org/10.1075/is.18.3.08zla
A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual (only body movement) and multimodal (body movement accompanied by nonlinguistic vocalization). Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario.
Article outline
  • 1.Introduction
  • 2.A “multimodal” origin of language?
    • 2.1Definitional issues: What is linguistic multimodality?
    • 2.2Theoretical issues: What are the alternatives?
    • 2.3Empirical issues: How do we decide?
  • 3.Pantomiming events with and without vocalization: An experimental study
    • 3.1Rationale and hypotheses
    • 3.2Materials
    • 3.3Participants and procedure
    • 3.4Results
    • 3.5Discussion
  • 4.Conclusions
  • Acknowledgements
  • Notes
  • References
References (59)
References
Aboitiz, F. (2012). Gestures, vocalizations, and memory in language origins. Frontiers in Evolutionary Neuroscience, 4(2), 1–15. Google Scholar logo with link to Google Scholar
Andrén, M. (2010). Children's gestures between 18 and 30 months. Lund: Media Tryck.Google Scholar logo with link to Google Scholar
Arbib, M. (2005). From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and brain sciences, 281, 105–168. Google Scholar logo with link to Google Scholar
(2012). How the brain got language. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Armstrong, D. F., Stokoe, W. C., & Wilcox, S. E. (1995). Gesture and the nature of language. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Armstrong, D. F., & Wilcox, S. E. (2007). The gestural origin of language. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Brown, J. E. (2012). The evolution of symbolic communication: An embodied perspective. PhD Thesis, University of Edinburgh.
Burling, R. (2005). The talking ape: How language evolved. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Call, J., & Tomasello, M. (2007). The gestural communication of apes and monkeys. London: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Chandler, D. (2007). Semiotics: The basics (2nd ed.). London: Routledge. Google Scholar logo with link to Google Scholar
Clay, Z., & Zuberbühler, K. (2014). Vocal communication and social awareness in chimpanzees and bonobos: Insights from studies of vocal communication. In D. Dor, C. Knight, & J. Lewis (Eds.), The social origins of language (pp. 141–156). Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Collins, C. (2013). Paleopoetics: The evolution of the literary imagination. New York: Columbia University Press.Google Scholar logo with link to Google Scholar
Corballis, M. C. (2003). From mouth to hand: Gesture, speech, and the evolution of right-handedness. Behavioral and Brain Sciences, 26(2), 199–208. Google Scholar logo with link to Google Scholar
Donald, M. (1991). Origins of the modern mind: Three stages in the evolution of human culture. Cambridge, MA: Harvard University Press.Google Scholar logo with link to Google Scholar
Dor, D. (2015). The instruction of imagination: Language as a social communication technology. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Dunbar, R. (1996). Grooming, gossip and the evolution of language. London: Faber & Faber.Google Scholar logo with link to Google Scholar
Fay, N., Arbib, M., & Garrod, D. (2013). How to bootstrapp a human communication system. Cognitive Science, 371, 1356–1367. Google Scholar logo with link to Google Scholar
Fay, N., Lister, C. J., Ellison, T. M., & Goldin-Meadow, S. (2014). Creating a communication system from scratch: gesture beats vocalization hands down. Frontiers in Psychology, 5, 354. Google Scholar logo with link to Google Scholar
Fitch, W. T. (2010). The evolution of language. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
(2011). Experimental semiotics: A review. Frontiers in Human Neuroscience, 5(11). Google Scholar logo with link to Google Scholar
Gardner, R. A., & Gardner, B. T. (1969). Teaching sign language to a chimpanzee. Science 1651, 664–672. Google Scholar logo with link to Google Scholar
Hewes, G. W. (1973). Primate communication and the gestural origin of language. Current Anthropology, 14(1/2), 5–24. Google Scholar logo with link to Google Scholar
Hobaiter, C. & Byrne, R. W. (2012). Gesture use in consortship: Wild chimpanzees’ use of gesture for an ‘evolutionarily urgent’ purpose. In S. Pika & K. Liebal (Eds.), Developments in primate gesture research (pp. 129–146). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Kemmerer, D., Chandrasekaran, B., & Tranel, D. (2007). A case of impaired verbalization but preserved gesticulation of motion events. Cognitive neuropsychology, 24(1), 70–114. Google Scholar logo with link to Google Scholar
Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
(2014a). Semiotic diversity in utterance production and the concept of “language”. Philosophical Transactions of the Royal Society B, 369(1651), 20130293. Google Scholar logo with link to Google Scholar
(2014b). The 'poly-modalic' nature of utterances and its implication. In D. Dor, C. Knight, & D. Lewis (Eds.), The social origins of language (pp. 67–76). Oxford: OUP.Google Scholar logo with link to Google Scholar
Kita, S., & Özyurek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 481, 16–32. Google Scholar logo with link to Google Scholar
Landau, B., & Gleitman, L. R. (2009). Language and experience: Evidence from the blind child. Cambridge, MA: Harvard University Press.Google Scholar logo with link to Google Scholar
Levinson, S. C., & Holler, J. (2014). The origin of human multi-modal communication. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 3691, 201303302. Google Scholar logo with link to Google Scholar
Lock, A., & Zukow-Goldring, P. (2010). Preverbal communication. In J. Bremner & T. Wachs (Eds.), The Wiley-Blackwell handbook of infant development (pp. 394–425). Oxford: Willey-Blackwell. Google Scholar logo with link to Google Scholar
MacNeilage, P. F. (2008). The origin of speech. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
McNeill, D. (1985). So you think gestures are nonverbal? Psychological Review, 921, 350–371. Google Scholar logo with link to Google Scholar
(2005). Gesture and thought. Chicago: University of Chicago Press. Google Scholar logo with link to Google Scholar
(2012). How language began: Gesture and speech in human evolution. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Meir, I., Aronoff, M., Börstell, C., Hwang, S., Ilkbasaran, D., Kastner, I., Lepic, R., Ben Basat, A. L., Padden, C., & Sandler, W. (2017). The effect of being human and the basis of grammatical word order: Insights from novel communication systems and young sign languages. Cognition, 1581, 189–207. Google Scholar logo with link to Google Scholar
Mithen, S. (2005). The singing Neanderthals: The origins of music, language, mind and body. London: Weidenfeld and Nicholson.Google Scholar logo with link to Google Scholar
Murillo, E., & Belinchón, M. (2012). Gestural-vocal coordination. Gesture, 121, 16–39. Google Scholar logo with link to Google Scholar
Paus, T. (2001). Primate anterior cingulate cortex: Where motor control, drive and cognition interface. Nature Reviews Neuroscience, 2(6), 417–424. Google Scholar logo with link to Google Scholar
Pika, S. (2008). What is the nature of gestural communication in great apes? The shared mind: Perspectives on intersubjectivity (pp. 165–186). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Schel, A. M., Townsend, S. W., Machanda, Z., Zuberbühler, K., & Slocombe, K. E. (2013). Chimpanzee alarm call production meets key criteria for intentionality. PloS One, 8(10), e76674. Google Scholar logo with link to Google Scholar
See, A. (2014). Reevaluating chimpanzee vocal signals: Toward a multimodal account of the origins of human communication. In M. Pina & N. Gontier (Eds.), The evolution of social communication in primates (pp. 195–215). New York: Springer. Google Scholar logo with link to Google Scholar
Sonesson, G. (1997). The ecological foundations of iconicity. Semiotics around the world: Synthesis in diversity (pp. 739–742). Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Stokoe, W. C. (1991). Semantic phonology. Sign Language Studies, 711, 107–114. Google Scholar logo with link to Google Scholar
Taglialatela, J. P., Russell, J. L., Schaeffer, J. A., & Hopkins, W. D. (2011). Chimpanzee vocal signaling points to a multimodal origin of human language. PloS One, 6(4), e18852. Google Scholar logo with link to Google Scholar
Tomasello, M. (2008). The origins of human communication. Cambridge, MA: MIT Press. Google Scholar logo with link to Google Scholar
Vigliocco, G., Perniss, P., & Vinson, D. (2014). Language as a multimodal phenomenon: implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society B, 369(1651), 20130292. Google Scholar logo with link to Google Scholar
Volterra, V., Caselli, M., Caprici, O., & Pizzuto, E. (2005). Gesture and the emergence and development of language. In M. Tomasello & D. Slobin (Eds.), Beyond nature-nurture: Essays in honor of Elisabeth Bates (pp. 3–40). Mahwah, NJ: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Wittgenstein, L. (1953). Philosophical investigations. Oxford: Blackwell.Google Scholar logo with link to Google Scholar
Zlatev, J. (1997). Situated embodiment: Studies in the emergence of spatial meaning. Stockholm: Gotab.Google Scholar logo with link to Google Scholar
(2008). From proto-mimesis to language: Evidence from primatology and social neuroscience. Journal of Physiology – Paris, 1021, 137–152. Google Scholar logo with link to Google Scholar
(2012). Cognitive semiotics: An emerging field for the transdisciplinary study of meaning. Public Journal of Semiotics, 41, 2–24. Google Scholar logo with link to Google Scholar
(2014). Bodily mimesis and the transition to speech. In M. Pina & N. Gontier (Eds.), The evolution of social communication in primates (pp. 165–178). New York: Springer. Google Scholar logo with link to Google Scholar
Zywiczynski, P., & Wacewicz, S. (2015). Ewolucja języka. W stronę hipotez gesturalnych. Toruń: Wydawnictwo Naukowe Uniwersytetu Mikołaja Kopernika.Google Scholar logo with link to Google Scholar
Zywiczynski, P., Wacewicz, S., & Sibierska, M. (2016). Defining pantomime for language evolution research Topoi. An International Review of Philosophy [in press]. Google Scholar logo with link to Google Scholar
Cited by (29)

Cited by 29 other publications

Gong, Siyi, Kaiwen Jiang, Jessica G. Li, Mireille Karadanaian, Ziyi Meng & Tao Gao
2025. Communicating Through Acting: Affording Communicative Intention in Pantomimes. Cognitive Science 49:10 DOI logo
Kuleshova, Svetlana & Michael Pleyer
2025. From extant to extinct: The role of experiments and interdisciplinary inferences in studying cognitive and language evolution. Journal of Archaeological Science 184 ► pp. 106423 ff. DOI logo
Nölle, Jonas M., Andres Karjus & Stefan Hartmann
2025. Evolutionary Linguistics. In Reference Module in Social Sciences, DOI logo
Podlipniak, Piotr
2025. The evolution of musicality and cross-domain co-evolutionary interactions. Musicae Scientiae 29:1 ► pp. 5 ff. DOI logo
Walker, Bradley & Nicolas Fay
2025. Gesture and the Origin of Language: Insights From Experimental Semiotics. In Reference Module in Social Sciences, DOI logo
Brown, Steven
2024. The pantomimic origins of the narrative arts. In Perspectives on Pantomime [Advances in Interaction Studies, 12], ► pp. 139 ff. DOI logo
Brown, Steven
2025. Role playing in human evolution: from life to art, and everything in between. Frontiers in Psychology 15 DOI logo
MacNeil, Raymond R. & James T. Enns
2024. The “What” and “How” of Pantomime Actions. Vision 8:4 ► pp. 58 ff. DOI logo
Źywiczyński, Przemysław & Jordan Zlatev
2024. The Role of Gesture in Debates on the Origins of Language. In The Cambridge Handbook of Gesture Studies, ► pp. 335 ff. DOI logo
Żywiczyński, Przemysław, Marek Placiński, Marta Sibierska, Monika Boruta-Żywiczyńska, Sławomir Wacewicz, Michał Meina & Peter Gärdenfors
2024. Praxis, demonstration and pantomime: a motion capture investigation of differences in action performances. Language and Cognition 16:4 ► pp. 1338 ff. DOI logo
Delliponti, Angelo, Renato Raia, Giulia Sanguedolce, Adam Gutowski, Michael Pleyer, Marta Sibierska, Marek Placiński, Przemysław Żywiczyński & Sławomir Wacewicz
2023. Experimental Semiotics: A Systematic Categorization of Experimental Studies on the Bootstrapping of Communication Systems. Biosemiotics 16:2 ► pp. 291 ff. DOI logo
Markov, Ilia, Kseniia Kharitonova & Elena L. Grigorenko
2023. Language: Its Origin and Ongoing Evolution. Journal of Intelligence 11:4 ► pp. 61 ff. DOI logo
Placiński, Marek & Przemysław Żywiczyński
2023. Modality effect in interactive alignment: Differences between spoken and text-based conversation. Lingua 293 ► pp. 103592 ff. DOI logo
Placiński, Marek, Przemysław Żywiczyński, Theresa Matzinger, Marta Sibierska, Monika Boruta-Żywiczyńska, Anna Szala & Sławomir Wacewicz
2023. Evolution of Pantomime in Dyadic Interaction. A Motion Capture Study. Journal of Language Evolution 8:2 ► pp. 134 ff. DOI logo
Sibierska, Marta, Przemysław Żywiczyński, Jordan Zlatev, Joost van de Weijer & Monika Boruta-Żywiczyńska
2023. Constraints on communicating the order of events in stories through pantomime. Journal of Language Evolution 8:1 ► pp. 18 ff. DOI logo
Holler, Judith
2022. Visual bodily signals as core devices for coordinating minds in interaction. Philosophical Transactions of the Royal Society B 377:1859 DOI logo
Rasenberg, Marlou, Asli Özyürek, Sara Bögels & Mark Dingemanse
2022. The Primacy of Multimodal Alignment in Converging on Shared Symbols for Novel Referents. Discourse Processes 59:3 ► pp. 209 ff. DOI logo
Sibierska, Marta, Monika Boruta-Żywiczyńska, Przemysław Żywiczyński & Sławomir Wacewicz
2022. What’s in a mime?. Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems 23:2 ► pp. 289 ff. DOI logo
Sonesson, Göran H.
2022. The Phenomenology of Semiosis: Approaches to the Gap between the Encyclopaedia and the Porphyrian Tree Spanned by Sedimentation. Philosophies 7:5 ► pp. 114 ff. DOI logo
Żywiczyński, Przemysław & Sławomir Wacewicz
2022. From Body to Language: Gestural and Pantomimic Scenarios of Language Origin in the Enlightenment. Topoi 41:3 ► pp. 539 ff. DOI logo
Ferretti, Francesco & Ines Adornetti
2021. Persuasive conversation as a new form of communication in Homo sapiens . Philosophical Transactions of the Royal Society B 376:1824 DOI logo
Olteanu, Alin
2021. Multimodal Modeling: Bridging Biosemiotics and Social Semiotics. Biosemiotics 14:3 ► pp. 783 ff. DOI logo
Żywiczyński, Przemysław, Marta Sibierska, Sławomir Wacewicz, Joost van de Weijer, Francesco Ferretti, Ines Adornetti, Alessandra Chiera & Valentina Deriu
2021. Evolution of conventional communication. A cross-cultural study of pantomimic re-enactments of transitive events. Language & Communication 80 ► pp. 191 ff. DOI logo
Żywiczyński, Przemysław, Sławomir Wacewicz & Casey Lister
2021. Pantomimic fossils in modern human communication. Philosophical Transactions of the Royal Society B 376:1824 DOI logo
Macuch Silva, Vinicius, Judith Holler, Asli Ozyurek & Seán G. Roberts
2020. Multimodality and the origin of a novel communication system in face-to-face interaction. Royal Society Open Science 7:1 ► pp. 182056 ff. DOI logo
Adornetti, Ines, Francesco Ferretti, Alessandra Chiera, Slawomir Wacewicz, Przemysław Żywiczyński, Valentina Deriu, Andrea Marini, Rita Magni, Laura Casula, Stefano Vicari & Giovanni Valeri
2019. Do Children With Autism Spectrum Disorders Understand Pantomimic Events?. Frontiers in Psychology 10 DOI logo
Brown, Steven, Emma Mittermaier, Tanishka Kher & Paul Arnold
2019. How Pantomime Works: Implications for Theories of Language Origin. Frontiers in Communication 4 DOI logo
Żywiczyński, Przemysław
2019. How research on language evolution contributes to linguistics. Yearbook of the Poznan Linguistic Meeting 5:1 ► pp. 1 ff. DOI logo
Zywiczynski, Przemyslaw, Nathalie Gontier & Slawomir Wacewicz
2017. The evolution of (proto-)language: Focus on mechanisms. Language Sciences 63 ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

video

Item 14. boy-push-man

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue