Cover not available

In:The Documentarist Turn: From observable linguistic behaviour to typological generalizations
Edited by Sonja Riesberg, Uta Reinöhl and Birgit Hellwig
[Studies in Language Companion Series 240] 2026
► pp. 210–238

Chapter 9
The multimodal turn in corpus building

Lessons from language documentation and sign language corpora

ORCiD logo with link | University of Cologne
ORCiD logo with link | University of Cologne
ORCiD logo with link | University of Cologne
ORCiD logo with link | University of Cologne
ORCiD logo with link | University of Cologne
This content is being prepared for publication; it may be subject to changes.
Abstract
This chapter argues for a decisive shift in corpus-building practices toward a Multimodal Turn. For decades, smaller-scale spoken and signed languages were less available for linguistic corpus research due to the scarcity of corpora, but the situation has changed: language documentation now provides rich multimodal resources, and many sign languages are recorded in high-definition video, offering unique insights into communicative practices. In contrast, corpora of majority languages often remain limited in video data. We contend that they can draw inspiration from the practices developed in language documentation and sign language research to close this gap. Only by systematically integrating multimodal interaction into corpus design can linguistic research capture the full complexity of human communication.
Article outline
  • 1.Introduction
  • 2.Multimodality and why it matters for language research
    • 2.1What is multimodality?
    • 2.2Investigating multimodal feedback across languages
  • 3.Types of language corpora
    • 3.1Language documentation corpora
    • 3.2Sign language corpora
    • 3.3Majority language corpora
  • 4.Multimodal corpus building
    • 4.1“A multipurpose record”: Multimodal data for cross-linguistic research
    • 4.2“A lasting record”: Archiving as an integral part of language research
    • 4.3Beyond the “observable”: Incorporating physiological data
  • 5.Concluding remarks
  • Author statement
  • Acknowledgments
  • Note
  • References
References (138)
References
Abner, Natasha, Kensy Cooperrider, and Susan Goldin-Meadow. 2015. “Gesture for Linguists: A Handy Primer.” Language and Linguistics Compass 9 (11): 437–451. Google Scholar logo with link to Google Scholar
Abuczki, Ágnes, and Esfandiari Baiat Ghazaleh. 2013. “An Overview of Multimodal Corpora, Annotation Tools and Schemes.” Argumentum 9: 86–98.Google Scholar logo with link to Google Scholar
Adolphs, Svenja, and Ronald Carter. 2013. Spoken Corpus Linguistics: From Monomodal to Multimodal. New York and London: Routledge. Google Scholar logo with link to Google Scholar
Allwood, Jens. 2008. “Multimodal Corpora.” In Corpus Linguistics: An International Handbook, Vol. 1, ed. by Anke Lüdeling, and Merja Kytö, 207–225. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Allwood, Jens, and Loredana Cerrato. 2003. “A Study of Gestural Feedback Expressions.” In First Nordic Symposium on Multimodal Communication, ed. by Patrizia Paggio, Kristiina Jokinen, and Arne Jönsson, 7–22. Copenhagen: Gothenburg University Publications.Google Scholar logo with link to Google Scholar
Ameka, Felix K., and Marina Terkourafi. 2019. “What If…? Imagining Non-Western Perspectives on Pragmatic Theory and Practice.” Journal of Pragmatics 145: 72–82. Google Scholar logo with link to Google Scholar
Auer, Peter, and Barbara Laner. 2025. “Laughter and Gaze among Talkers on a Walk.” In Mobile Eye Tracking: New Avenues for the Study of Gaze in Social Interaction, ed. by Elisabeth Zima, and Anja Stukenbrock, 208–242. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Badcock, Nicholas A., Petroula Mousikou, Yatin Mahajan, Peter de Lissa, Johnson Thie, and Genevieve McArthur. 2013. “Validation of the Emotiv EPOC® EEG Gaming System for Measuring Research Quality Auditory ERPs.” PeerJ 1: e38. Google Scholar logo with link to Google Scholar
Bauer, Anastasia. 2014. The Use of Signing Space in a Shared Sign Language of Australia. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
. 2023. Russian Multimodal Conversational Corpus. Data Center for the Humanities, University of Cologne. Google Scholar logo with link to Google Scholar
Bauer, Anastasia, Sonja Gipper, Tobias-Alexander Herrmann, and Jana Hosemann. 2026. “Rethinking Linguistic Feedback: A Modality-Agnostic and Holistic Approach to Multimodal Addressee Signals in Spoken and Signed Dyadic Interaction.” Glossa: a journal of general linguistics 11 (1). Google Scholar logo with link to Google Scholar
Bauer, Anastasia, Anna Kuder, Marc Schulder, and Job Schepens. 2024. “Phonetic Differences between Affirmative and Feedback Head Nods in German Sign Language (DGS): A Pose Estimation Study.” PLoS ONE 19 (5): e0304040. Google Scholar logo with link to Google Scholar
Bauer, Anastasia, and Liona Paulus. To appear. “The Multimodality of Sign Languages.” In The Handbook of Linguistics and Multimodality, ed. by John Bateman, and Chiao-I. Tseng. Hoboken, NJ: Wiley.
Bauer, Anastasia, and Roman Poryadin. 2023. Russian Sign Language Conversations. Data Center for the Humanities, University of Cologne. Google Scholar logo with link to Google Scholar
Bauer, Anastasia, Patrick C. Trettenbrein, Federica Amici, Aleksandra Ćwiek, Susanne Fuchs, Lisa-Marie Krause, Anna Kuder, Silva Ladewig, Marc Schulder, Petra Schumacher, Door Spruijt, Chiara Zulberti, and Martin Schulte-Rüther. 2025. “Data Collection in Multimodal Language and Communication Research: A Flexible Decision framework.” Preprint, OSF. Google Scholar logo with link to Google Scholar
Bavelas, Janet, Nicole Chovil, Linda Coates, and Lori Roe. 1995. “Gestures Specialized for Dialogue.” Personality and Social Psychology Bulletin 21 (4): 394–405. Google Scholar logo with link to Google Scholar
Bavelas, Janet, Linda Coates, and Trudy Johnson. 2000. “Listeners as Co-Narrators.” Journal of Personality and Social Psychology 79 (6): 941–952. Google Scholar logo with link to Google Scholar
BNC Consortium. 2007. The British National Corpus XML Edition. [URL]
Borrego, Adrian, Jorge Latorre, Mariano Alcaniz, and Roberto Llorens. 2019. “Reliability of the Empatica E4 Wristband to Measure Electrodermal Activity to Emotional Stimuli.” In Proceedings of the 2019 International Conference on Virtual Rehabilitation (ICVR), 1–2. Piscataway, NJ: Institute of Electrical and Electronics Engineers. Google Scholar logo with link to Google Scholar
Brunner, Lawrence J. 1979. “Smiles Can Be Back Channels.” Journal of Personality and Social Psychology 37 (5): 728–734. Google Scholar logo with link to Google Scholar
Burkova, Svetlana. 2015. Russian Sign Language Corpus. Novosibirsk University. [URL]
Cao, Zhe, Ginés Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A. Sheikh. 2021. “OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields.” IEEE Transactions on Pattern Analysis and Machine Intelligence 43 (1): 172–186. Google Scholar logo with link to Google Scholar
Chafe, Wallace. 1992. “The Importance of Corpus Linguistics to Understanding the Nature of Language.” In Directions in Corpus Linguistics: Proceedings of Nobel Symposium 82 Stockholm, 4–8 August 1991, ed. by Jan Svartvik, 79–104. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Chatterjee, Iman, Maja Goršič, Joshua D. Clapp, and Domen Novak. 2021. “Automatic Estimation of Interpersonal Engagement during Naturalistic Conversation Using Dyadic Physiological Measurements.” Frontiers in Neuroscience 15: 757381. Google Scholar logo with link to Google Scholar
Chen, Phoebe, Ulrich Kirk, and Suzanne Dikker. 2022. “Trait Mindful Awareness Predicts Inter-Brain Coupling but Not Individual Brain Responses during Naturalistic Face-to-Face Interactions.” Frontiers in Psychology 13: 915345. Google Scholar logo with link to Google Scholar
Chen, Yaoyao, and Svenja Adolphs. 2023. “Towards a Corpus-Based Description of Speech-Gesture Units of Meaning: The Case of the Circular Gesture.” International Journal of Corpus Linguistics 28 (2): 172–201. Google Scholar logo with link to Google Scholar
Cohn, Neil, and Joost Schilperoord. 2024. A Multimodal Language Faculty: A Cognitive Framework for Human Communication. London, New York and Dublin: Bloomsbury. Google Scholar logo with link to Google Scholar
Conklin, Kathy, Ana Pellicer-Sánchez, and Gareth Carrol. 2018. Eye-Tracking: Applied Linguistics and Second Language Acquisition. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Davies, Mark. 2008. The Corpus of Contemporary American English (COCA). [URL]
Davis, Jeffrey. 2010. Hand Talk: Sign Language Among American Indian Nations. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Dikker, Suzanne, Georgios Michalareas, Matthias Oostrik, Amalia Serafimaki, Hasibe Melda Kahraman, Marijn E. Struiksma, and David Poeppel. 2021. “Crowdsourcing Neuroscience: Inter-Brain Coupling during Face-to-Face Interactions Outside the Laboratory.” NeuroImage 227: 117436. Google Scholar logo with link to Google Scholar
Dimigen, Olaf, and Benedikt V. Ehinger. 2021. “Regression-Based Analysis of Combined EEG and Eye-Tracking Data: Theory and Applications.” Journal of Vision 21 (1): 3. Google Scholar logo with link to Google Scholar
Dingemanse, Mark, and Nick J. Enfield. 2015. “Other-Initiated Repair across Languages: Towards a Typology of Conversational Structures.” Open Linguistics 1 (1): 96–118. Google Scholar logo with link to Google Scholar
Dittmann, Allen T., and Lynn G. Llewellyn. 1968. “Relationship between Vocalizations and Head Nods as Listener Responses.” Journal of Personality and Social Psychology 9 (1): 79–84. Google Scholar logo with link to Google Scholar
Dobrushina, Nina, and Elena Sokur. 2022. “Spoken Corpora of Slavic Languages.” Russian Linguistics 46 (2): 77–93. Google Scholar logo with link to Google Scholar
Enfield, Nick J. 2009. The Anatomy of Meaning: Speech, Gesture, and Composite Utterances. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Ernestus, Mirjam. 2000–2024. Nijmegen Corpora of Casual Speech. The Language Archive, MPI Nijmegen. [URL]
Fenlon, Jordan, Adam Schembri, Trevor Johnston, and Kearsy Cormier. 2015. “Documentary and Corpus Approaches to Sign Language Research.” In Research Methods in Sign Language Studies, ed. by Eleni Orfanidou, Bencie Woll, and Gary Morgan, 156–172. Chichester: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Ferrara, Lindsay, and Gabrielle Hodge. 2018. “Language as Description, Indication, and Depiction.” Frontiers in Psychology 9: 716. Google Scholar logo with link to Google Scholar
Gijn, Rik van, Vincent Hirtzel, Sonja Gipper, and Jeremías Ballivián Torrico. 2011. The Yurakaré Archive. The Language Archive, MPI Nijmegen. [URL]
Gipper, Sonja, Eugen Hill, and Pascal Coenen. This volume. “Language Change in Real Time: Investigating Conversational Priming in Repetitional Responses.” In The Documentarist Turn: From Observable Linguistic Behaviour to Typological Generalizations, ed. by Sonja Riesberg, Uta Reinöhl, and Birgit Hellwig. Amsterdam: John Benjamins.
Gipper, Sonja, Anastasia Bauer, Alice Mitchell, and Alena Witzlack-Makarevich. 2025. “Cross-Linguistic Variation in Multimodal Feedback in Conversation.” Presentation, 24. Arbeitstagung zur Gesprächsforschung, Mannheim, 19–21 March 2025.
Green, Jennifer. 2014. Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Gregori, Alina, Federica Amici, Ingmar Brilmayer, Aleksandra Ćwiek, Lennart Fritzsche, Susanne Fuchs, Alexander Henlein, Oliver Herbort, Frank Kügler, Jens Lemanski, Katja Liebal, Andy Lücking, Alexander Mehler, Kim Tien Nguyen, Wim Pouw, Pilar Prieto, Patrick Louis Rohrer, Paula G. Sánchez-Ramón, Martin Schulte-Rüther, Petra B. Schumacher, Stefan R. Schweinberger, Volker Struckmeier, Patrick C. Trettenbrein, and Celina I. von Eiff. 2023. “A Roadmap for Technological Innovation in Multimodal Communication Research.” In Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management, ed. by Vincent G. Duffy, 402–438. Cham: Springer. Google Scholar logo with link to Google Scholar
Grišina, Elena. 2017. Russkaja žestikuljacija s lingvističeskoj točki zrenija: Korpusnye issledovanija [Russian gesticulation from a linguistic point of view: Corpus-based studies]. Moskva: Izdatel’skij dom JASK (Razumnoe povedenie).Google Scholar logo with link to Google Scholar
Hagoort, Peter, and Aslı Özyürek. 2025. “Extending the Architecture of Language from a Multimodal Perspective.” Topics in Cognitive Science Online First: 12728. Google Scholar logo with link to Google Scholar
Hale, Ken. 1998. “On Endangered Languages and the Importance of Linguistic Diversity.” In Endangered Languages: Language Loss and Community Response, ed. by Lenore A. Grenoble, and Lindsay J. Whaley, 192–216. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Hamilton, Antonia F. de C., and Judith Holler. 2023. “Face2face: Advancing the Science of Social Interaction.” Philosophical Transactions of the Royal Society B: Biological Sciences 378 (1875): 20210470. Google Scholar logo with link to Google Scholar
Hanke, Thomas. 2014. “Annotation of Mouth Activities with iLex.” In Proceedings of the 6th Workshop on the Representation and Processing of Sign Languages: Beyond the Manual Channel (LREC 2014), ed. by Onno Crasborn, Eleni Efthimiou, Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen, and Johanna Mesch, 67–70. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Hanke, Thomas, Marc Schulder, Reiner Konrad, and Elena Jahn. 2020. “Extending the Public DGS Corpus in Size and Depth.” In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), ed. by Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen, and Johanna Mesch, 75–82. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Hardacre, Bahiyyih. 2020. Psychophysiological Methods in Language Research: Rethinking Embodiment in Studies of Linguistic Behaviors. Lanham, MD: Lexington. Google Scholar logo with link to Google Scholar
Hayes, Hannah Begue, and Cyrille Magne. 2024. “Exploring the Utility of the Muse Headset for Capturing the N400: Dependability and Single-Trial Analysis.” Sensors 24 (24): 7961. Google Scholar logo with link to Google Scholar
Helm, Jonathan L., Jonas G. Miller, Sarah Kahle, Natalie R. Troxel, and Paul D. Hastings. 2018. “On Measuring and Modeling Physiological Synchrony in Dyads.” Multivariate Behavioral Research 53 (4): 521–543. Google Scholar logo with link to Google Scholar
Himmelmann, Nikolaus P. 1998. “Documentary and Descriptive Linguistics.” Linguistics 36 (1): 161–196. Google Scholar logo with link to Google Scholar
2006. “Language Documentation: What Is It and What Is It Good for? In Essentials of Language Documentation, ed. by Jost Gippert, Nikolaus P. Himmelmann, and Ulrike Mosel, 1–30. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
2008. “Reproduction and Preservation of Linguistic Knowledge: Linguistics’ Response to Language Endangerment.” Annual Review of Anthropology 37 (1): 337–350. Google Scholar logo with link to Google Scholar
2012. “Linguistic Data Types and the Interface between Language Documentation and Description.” Language Documentation and Conservation 6: 187–207. [URL]
Hochgesang, Julie A., Ryan Lepic, and Emily Shaw. 2023. “W(h)ither the ASL Corpus? Considering Trends in Signed Corpus Development.” In Advances in Sign Language Corpus Linguistics, ed. by Ella Wehrmeyer, 287–308. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Hoffmann, Bettina, and Nikolaus P. Himmelmann. 2009. “Münster Videokorpus Alltagsgespräche [Münster video corpus of everyday conversations].” Unpublished corpus.Google Scholar logo with link to Google Scholar
Holler, Judith, and Stephen C. Levinson. 2019. “Multimodal Language Processing in Human Communication.” Trends in Cognitive Sciences 23 (8): 639–652. Google Scholar logo with link to Google Scholar
Hömke, Paul, Stephen C. Levinson, Alexandra K. Emmendorfer, and Judith Holler. 2025. “Eyebrow Movements as Signals of Communicative Problems in Human Face-to-Face Interaction.” Royal Society Open Science 12: 12241632. Google Scholar logo with link to Google Scholar
Hosemann, Jana. 2013. “Eye Gaze and Verb Agreement in German Sign Language: A First Glance.” In Nonmanuals in Sign Language, ed. by Annika Herrmann, and Markus Steinbach, 73–90. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Jefferson, Gail. 1984. “Notes on a Systematic Deployment of the Acknowledgement Tokens ‘Yeah’; and ‘Mm Hm’;” Paper in Linguistics 17 (2): 197–216. Google Scholar logo with link to Google Scholar
Johnston, Trevor. 2008. Auslan Corpus. Endangered Languages Archive. [URL]
Johnston, Trevor, and Adam Schembri. 2013. “Corpus Analysis of Sign Languages.” In The Encyclopedia of Applied Linguistics, ed. by Carol A. Chapell, 479–501. Chichester: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Joyce, Jack B., Tom Douglass, Bethan Benwell, Catrin S. Rhys, Ruth Parry, Richard Simmons, and Adrian Kerrison. 2022. “Should We Share Qualitative Data? Epistemological and Practical Insights from Conversation Analysis.” International Journal of Social Research Methodology 26 (6): 645–659. Google Scholar logo with link to Google Scholar
Just, Marcel A., and Patricia A. Carpenter. 1980. “A Theory of Reading: From Eye Fixations to Comprehension.” Psychological Review 87 (4): 329–354. Google Scholar logo with link to Google Scholar
Keevallik, Leelo. 2018. “What Does Embodied Interaction Tell Us about Grammar?” Research on Language and Social Interaction 51 (1): 1–21. Google Scholar logo with link to Google Scholar
Kendon, Adam. 1988. Sign Languages of Aboriginal Australia: Cultural, Semiotic and Communicative Perspectives. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kendrick, Kobin H., Judith Holler, and Stephen C. Levinson. 2023. “Turn-Taking in Human Face-to-Face Interaction Is Multimodal: Gaze Direction and Manual Gestures Aid the Coordination of Turn Transitions.” Philosophical Transactions of the Royal Society B: Biological Sciences 378 (1875): 20210473. Google Scholar logo with link to Google Scholar
Knight, Dawn, and Svenja Adolphs. 2020. “Multimodal Corpora.” In A Practical Handbook of Corpus Linguistics, ed. by Magali Paquot, and Stefan Th. Gries, 353–371. Cham: Springer. Google Scholar logo with link to Google Scholar
Konrad, Reiner, Thomas Hanke, Gabriele Langer, Dolly Blanck, Julian Bleicken, Ilona Hofmann, Olga Jeziorski, Lutz König, Susanne König, Rie Nishio, Anja Regen, Uta Salden, Sven Wagner, Satu Worseck, and Marc Schulder. 2020. MY DGS — Annotated. Public Corpus of German Sign Language. Release 3.0. Google Scholar logo with link to Google Scholar
Kopf, Maria, Marc Schulder, and Thomas Hanke. 2024. The Sign Language Dataset Compendium. Tech. rep. Version 1.3.1. Hamburg University. Google Scholar logo with link to Google Scholar
Kozanitis, Anastassis. 2023. “Comparing Real Time Student Situational Engagement in Traditional and Active Learning Classroom Using Non-Invasive Electrodermal Measurements.” In Proceedings of the 14th International Multi-Conference on Complexity, Informatics and Cybernetics (IMCIC 2023), ed. by Nagib C. Callaos, Shigehiro Hashimoto, Natalja Lace, Belkis Sánchez, and Michael Savoie, 133–138. Winter Garden: International Institute of Informatics and Cybernetics. Google Scholar logo with link to Google Scholar
Krigolson, Olave E., Chad C. Williams, Angela Norton, Cameron D. Hassall, and Francisco L. Colino. 2017. “Choosing MUSE: Validation of a Low-Cost, Portable EEG System for ERP Research.” Frontiers in Neuroscience 11: 109. Google Scholar logo with link to Google Scholar
Kroczek, Leon O. H., Angelika Lingnau, Valentin Schwind, Christian Wolff, and Andreas Mühlberger. 2024. “Observers Predict Actions from Facial Emotional Expressions during Real-Time Social Interactions.” Behavioural Brain Research 471: 115126. Google Scholar logo with link to Google Scholar
Küttner, Uwe-A., Laurenz Kornfeld, Christina Mack, Lorenza Mondada, Jowita Rogowska, Giovanni Rossi, Marja-Leena Sorjonen, Matylda Weidner, and Jörg Zinken. 2024. “Introducing the ‘Parallel European Corpus of Informal Interaction’ (PECII): A Novel Resource for Exploring Cross-Situational and Cross-Linguistic Variability in Social Interaction.” In New Perspectives in Interactional Linguistic Research, ed. by Margret Selting, and Dagmar Barth-Weingarten, 132–160. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Lahat, Dana, Tülay Adali, and Christian Jutten. 2015. “Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects.” Proceedings of the IEEE 103 (9): 1449–1477. Google Scholar logo with link to Google Scholar
Levinson, Stephen C. 2006. “On the Human “Interaction Engine”.” In Roots of Human Sociality: Culture, Cognition and Interaction, ed. by Nick J. Enfield, and Stephen C. Levinson, 39–69. Oxford: Berg. Google Scholar logo with link to Google Scholar
Lindberg, Rachael, Kim McDonough, and Pavel Trofimovich. 2021. “Investigating Verbal and Nonverbal Indicators of Physiological Response during Second Language Interaction.” Applied Psycholinguistics 42 (6): 1403–1425. Google Scholar logo with link to Google Scholar
Liu, Difei, Shen Liu, Xiaoming Liu, Chong Zhang, Aosika Li, Chenggong Jin, Yijun Chen, Hangwei Wang, and Xiaochu Zhang. 2018. “Interactive Brain Activity: Review and Progress on EEG-based Hyperscanning in Social Interactions.” Frontiers in Psychology 9: 1862. Google Scholar logo with link to Google Scholar
Lutzenberger, Hannah, Lierin de Wael, Rehana Omardeen, and Mark Dingemanse. 2024. “Interactional Infrastructure across Modalities: A Comparison of Repair Initiators and Continuers in British Sign Language and British English.” Sign Language Studies 24 (3): 548–581. Google Scholar logo with link to Google Scholar
Macuch Silva, Vinicius, Judith Holler, Aslı Özyürek, and Seán G. Roberts. 2020. “Multimodality and the Origin of a Novel Communication System in Face-to-Face Interaction.” Royal Society Open Science 7 (1): 182056. Google Scholar logo with link to Google Scholar
Marzoratti, Analia, and Tanya M. Evans. 2022. “Measurement of Interpersonal Physiological Synchrony in Dyads: A Review of Timing Parameters Used in the Literature.” Cognitive, Affective, and Behavioral Neuroscience 22 (6): 1215–1230. Google Scholar logo with link to Google Scholar
Mazzini, Sara, Judith Holler, and Linda Drijvers. 2023. “Studying Naturalistic Human Communication Using Dual-EEG and Audio-Visual Recordings.” STAR Protocols 4 (3): 102370. Google Scholar logo with link to Google Scholar
Mondada, Lorenza. 2016. “Challenges of Multimodality: Language and the Body in Social Interaction.” Journal of Sociolinguistics 20 (3): 336–366. Google Scholar logo with link to Google Scholar
Müller, Cornelia, Silva H. Ladewig, and Jana Bressem. 2013. “Gestures and Speech from a Linguistic Perspective: A New Field and Its History.” In Body — Language — Communication: An International Handbook on Multimodality in Human Interaction, ed. by Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill, and Sedinha Tessendorf, 55–81. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Murata, Aiko, Hiroshi Nishida, Katsumi Watanabe, and Tatsuya Kameda. 2020. “Convergence of Physiological Responses to Pain during Face-to-Face Interaction.” Scientific Reports 10 (1): 450. Google Scholar logo with link to Google Scholar
Næss, Åshild, and Sebastian Sauppe. 2025. “Bringing Psycholinguistics to the Field: Experiences from Solomon Islands.” Language Documentation and Conservation 19: 99–119. [URL]
Negi, Shivsevak, and Ritayan Mitra. 2018. “EEG Metrics to Determine Cognitive Load and Affective States: A Pilot Study.” In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (UbiComp 2018), 182–185. New York: Association for Computing Machinery. Google Scholar logo with link to Google Scholar
Nyst, Victoria. 2007. A Descriptive Analysis of Adamorobe Sign Language (Ghana). Utrecht: LOT. [URL]
Offrede, Tom, Christine Mooshammer, Alessandro D’Ausilio, and Susanne Fuchs. 2024. “Speech Adaptation and Physiological Responses: A Study on f0 and Skin Temperature.” In Proceedings of the International Conference on Speech Prosody 2024, ed. by Yiya Chen, Aoju Chen, and Amalia Arvaniti, 467–471. International Speech Communication Association. Google Scholar logo with link to Google Scholar
Özyürek, Aslı. 2021. “Considering the Nature of Multimodal Language from a Crosslinguistic Perspective.” Journal of Cognition 4 (1): 42. Google Scholar logo with link to Google Scholar
Pagán Cánovas, Cristóbal, Javier Valenzuela, Daniel Alcaraz Carrión, Inés Olza, and Michael Ramscar. 2020. “Quantifying the Speech-Gesture Relation with Massive Multimodal Datasets: Informativity in Time Expressions.” PLoS ONE 15 (6): e0233892. Google Scholar logo with link to Google Scholar
Paggio, Patrizia, and Costanza Navarretta. 2013. “Head Movements, Facial Expressions and Feedback in Conversations: Empirical Evidence from Danish Multimodal Data. Journal on Multimodal User Interfaces 7 (1–2): 29–37. Google Scholar logo with link to Google Scholar
Partan, Sarah, and Peter Marler. 1999. “Communication Goes Multimodal.” Science 283 (5406): 1272–1273. Google Scholar logo with link to Google Scholar
Perniss, Pamela. 2018. “Why We Should Study Multimodal Language.” Frontiers in Psychology 9: 1109. Google Scholar logo with link to Google Scholar
Pfau, Roland. 2012. “Manual Communication Systems: Evolution and Variation.” In Sign Language: An International Handbook, ed. by Roland Pfau, Markus Steinbach, and Bencie Woll, 513–551. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Płużyczka, Monika. 2018. “The First Hundred Years: A History of Eye Tracking as a Research Method.” Applied Linguistics Papers 25 (4): 101–116. Google Scholar logo with link to Google Scholar
Pouw, Wim. 2024. EnvisionBOX Modules for Social Signal Processing. [URL]
Pouw, Wim, James P. Trujillo, and James A. Dixon. 2020. “The Quantification of Gesture–Speech Synchrony: A Tutorial and Validation of Multimodal Data Acquisition Using Device-Based and Video-Based Motion Tracking.” Behavior Research Methods 52 (2): 723–740. Google Scholar logo with link to Google Scholar
Prince, Kilu von, and Sebastian Nordhoff. 2020. “An Empirical Evaluation of Annotation Practices in Corpora from Language Documentation.” In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), ed. by Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, 2778–2787. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Rasenberg, Marlou, Aslı Özyürek, and Mark Dingemanse. 2020. “Alignment in Multimodal Interaction: An Integrative Framework.” Cognitive Science 44 (11): e12911. Google Scholar logo with link to Google Scholar
Rau, Felix. This volume. “Language Archives in Context.” In The Documentarist Turn: From Observable Linguistic Behaviour to Typological Generalizations, ed. by Sonja Riesberg, Uta Reinöhl, and Birgit Hellwig. Amsterdam: John Benjamins.
Ravelli, Louise. 2024. “Centering Multimodality: A Personal and Disciplinary Journey.” Multimodality and Society 4 (3): 259–274. Google Scholar logo with link to Google Scholar
Rees, Alice, Lewis Bott, and Petra B. Schumacher. 2019. “Event-Related Potentials in Pragmatic Priming.” Neuroscience Letters 712: 134435. Google Scholar logo with link to Google Scholar
Reineke, Silke, Arnulf Deppermann, and Thomas Schmidt. 2023. “Das Forschungs- und Lehrkorpus für Gesprochenes Deutsch (FOLK).” In Korpora in der germanistischen Sprachwissenschaft: Mündlich, schriftlich, multimedial, ed. by Arnulf Deppermann, Christian Fandrych, Marc Kupietz, and Thomas Schmidt, 71–102. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Rochet-Capellan, Amélie, and Susanne Fuchs. 2014. ”Take a Breath and Take the Turn: How Breathing Meets Turns in Spontaneous Dialogue.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1658): 20130399. Google Scholar logo with link to Google Scholar
Rohrer, Patrick Louis, Ronny Bujok, Lieke van Maastricht, and Hans Rutger Bosker. 2025. “From ‘I Dance’ to ‘She Danced’ with a Flick of the Hands: Audiovisual Stress Perception in Spanish.” Psychonomic Bulletin and Review 32 (5): 2136–2145. Google Scholar logo with link to Google Scholar
Rossano, Federico, Penelope Brown, and Stephen C. Levinson. 2009. “Gaze, Questioning, and Culture.” In Conversation Analysis: Comparative Perspectives, ed. by Jack Sidnell, 187–249. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Rühlemann, Christoph, and Mathias Barthel. 2024. “Word Frequency and Cognitive Effort in Turns-at-Talk: Turn Structure Affects Processing Load in Natural Conversation.” Frontiers in Psychology 15: 1208029. Google Scholar logo with link to Google Scholar
Rühlemann, Christoph, and Alexander Ptak. 2023. “Reaching Beneath the Tip of the Iceberg: A Guide to the Freiburg Multimodal Interaction Corpus.” Open Linguistics 9 (1): 20220245. Google Scholar logo with link to Google Scholar
Rühlemann, Christoph, and James P. Trujillo. 2024. “The Effect of Gesture Expressivity on Emotional Resonance in Storytelling Interaction.” Frontiers in Psychology 15: 1477263. Google Scholar logo with link to Google Scholar
Sandler, Wendy. 2022. “Redefining Multimodality.” Frontiers in Communication 6: 758993. Google Scholar logo with link to Google Scholar
. 2024. “Speech and Sign: The Whole Human Language.” Theoretical Linguistics 50 (1–2): 107–124. Google Scholar logo with link to Google Scholar
Sandler, Wendy, and Diane Lillo-Martin. 2006. Sign Language and Linguistic Universals. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Sarvasy, Hannah S., Adam Milton Morgan, Jenny Yu, Victor S. Ferreira, and Shota Momma. 2023. “Cross-Clause Planning in Nungon (Papua New Guinea): Eye-tracking Evidence.” Memory and Cognition 51 (3): 666–680. Google Scholar logo with link to Google Scholar
Satyawati, Ni Made Dadi Astini, Ni Made Sumarni, Ketut Kanta, Josefina Safar, Hannah Lutzenberger, Nick Palfreyman, and Connie de Vos. 2025. “The Balinese Homesign Corpus: New Insights into Corpus Development in a Rural Signing Context.” Preprint, OSF. Google Scholar logo with link to Google Scholar
Sauppe, Sebastian, Caroline Andrews, and Elisabeth Norcliffe. 2023. “Experimental Research in Cross-Linguistic Psycholinguistics.” In The Routledge Handbook of Experimental Linguistics, ed. by Sandrine Zufferey, and Pascal Gygax, 156–172. New York and London: Routledge. Google Scholar logo with link to Google Scholar
Savčuk, Svetlana O., and Alexandra Makhova. 2021. “Multimodal Russian Corpus and Its Use in Emotional Studies.” Russian Journal of Communication 13 (1): 42–61. Google Scholar logo with link to Google Scholar
Savčuk, Svetlana O., Timofej A. Arxangel’skij, Anastasija A. Bonč-Osmolovskaja, Ol’ga V. Donina, Julija N. Kuznecova, Ol’ga N. Ljaševskaja, Boris V. Orexov, and Marija V. Podrjadčikova. 2024. “Nacional’nyj korpus russkogo jazyka 2.0: novye vozmožnosti i perspektivy razvitija” [Russian National Corpus 2.0: New opportunities and development prospects]. Voprosy jazykoznanija 2024 (2): 7–34. Google Scholar logo with link to Google Scholar
Schegloff, Emanuel A. 1982. “Discourse as an Interactional Achievement: Some Uses of ‘uh huh’ and Other Things that Come between Sentences.” In Analyzing Discourse: Text and Talk, ed. by Deborah Tannen, 71–93. Washington, D.C.: Georgetown University Press.Google Scholar logo with link to Google Scholar
Schembri, Adam, Jordan Fenlon, Ramas Rentelis, and Kearsy Cormier. 2011. British Sign Language Corpus Project: A Corpus of Digital Video Data of British Sign Language 2008–2011. First Edition. [URL]
Schulder, Marc, and Thomas Hanke. 2020. OpenPose in the Public DGS Corpus. Version 2. Hamburg University. Google Scholar logo with link to Google Scholar
. 2022. “How to Be FAIR When You CARE: The DGS Corpus as a Case Study of Open Science Resources for Minority Languages.” In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), ed. by Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, 164–173. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Seyfeddinipur, Mandana, Pamela Perniss, and Claudia Wegener. This volume. “Taking Language Use Seriously: On the Importance of Multimodality and Interaction in Documentary Practices.” In The Documentarist Turn: From Observable Linguistic Behaviour to Typological Generalizations, ed. by Sonja Riesberg, Uta Reinöhl, and Birgit Hellwig. Amsterdam: John Benjamins.
Stephenson, Lisa J., S. Gareth Edwards, and Andrew P. Bayliss. 2021. “From Gaze Perception to Social Cognition: The Shared-Attention System.” Perspectives on Psychological Science 16 (3): 553–576. Google Scholar logo with link to Google Scholar
Vigliocco, Gabriella, Pamela Perniss, and David Vinson. 2014. “Language as a Multimodal Phenomenon: Implications for Language Learning, Processing and Evolution.” Philosophical Transactions of the Royal Society B: Biological Sciences 369 (1651): 20130292. Google Scholar logo with link to Google Scholar
Vos, Connie de, and Ulrike Zeshan. 2012. “Introduction: Demographic, Sociocultural, and Linguistic Variation across Rural Signing Communities.” In Sign Languages in Village Communities, ed. by Ulrike Zeshan, and Connie de Vos, 2–24. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Wiener, Seth, and Kiwako Ito. 2014. “Do Syllable-Specific Tonal Probabilities Guide Lexical Access? Evidence from Mandarin, Shanghai and Cantonese Speakers.” Language, Cognition and Neuroscience 30 (9): 1048–1060. Google Scholar logo with link to Google Scholar
Williams, Nikolas S., Genevieve M. McArthur, Bianca de Wit, George Ibrahim, and Nicholas A. Badcock. 2020. “A Validation of Emotiv EPOC Flex Saline for EEG and ERP Research.” PeerJ 8: e9713. Google Scholar logo with link to Google Scholar
Wittenburg, Peter, Hennie Brugman, Albert Russel, Alex Klassmann, and Han Sloetjes. 2006. “ELAN: A Professional Framework for Multimodality Research.” In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), ed. by Nicoletta Calzolari, Khalid Choukri, Aldo Gangemi, Bente Maegaard, Joseph Mariani, Jan Odijk, and Daniel Tapias, 1556–1559. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Yngve, Victor H. 1970. “On Getting a Word in Edgewise.” In Papers from the 6th Regional Meeting of the Chicago Linguistic Society, 567–577. Chicago, Illinois: University of Chicago.Google Scholar logo with link to Google Scholar
Zima, Elisabeth. 2017. “On the Multimodality of [all the way from X PREP Y].” Linguistics Vanguard 3 (s1): 20160055. Google Scholar logo with link to Google Scholar
Zima, Elisabeth, Peter Auer, and Christoph Rühlemann. 2025. “Why Research on Gaze in Social Interaction Needs Mobile Eye Tracking.” In Mobile Eye Tracking: New Avenues for the Study of Gaze in Social Interaction, ed. by Elisabeth Zima, and Anja Stukenbrock, 24–66. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue