Cover not available

In:Dynamism in Metaphor and Beyond
Edited by Herbert L. Colston, Teenie Matlock and Gerard J. Steen
[Metaphor in Language, Cognition, and Communication 9] 2022
► pp. 95–108

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Multimodal body, multimodal mind, multimodal communication

 | Institute Professor and Professor of Cognitive Science Case Western Reserve University
Published online: 9 September 2022
DOI logo with link
https://doi.org/10.1075/milcc.9.05tur
Abstract
This chapter reviews prospects for improving the scientific study of communication by developing big data infrastructure and computational, technical, and statistical tools. Such big data systems are typically free of the methodological threats to validity and underpowered datasets common in lab experiments.
Article outline
  • An illumination
  • The foundation of communication is multimodal
  • To know a communicative system is to know a relational network of form-meaning pairs and how they blend
  • Communication science and multimodal big data
  • An infrastructure for research on multimodal communication
  • Examples
  • Conclusion
  • References
References (32)
References
Alexander, James. (2011). Blending in mathematics. Semiotica. 2011(187), 1–48. ISSN (Online) 1613-3692, ISSN (Print) 0037-1998, Google Scholar logo with link to Google Scholar
Craddock, R. Cameron, Daniel S. Margulies, Pierre Bellec, B. Nolan Nichols, Sarael Alcauter, Fernando A. Barrios, Yves Burnod, et al. (2016). Brainhack: a collaborative workshop for the open neuroscience community. GigaScience 5:16. Google Scholar logo with link to Google Scholar
Davies, Mark. (2015). The importance of robust corpora in providing more realistic descriptions of variation in English grammar. Linguistics Vanguard 1(1). 305–312. Google Scholar logo with link to Google Scholar
Fauconnier, Gilles and Mark Turner. (1996). Blending as a Central Process of Grammar. In Conceptual Structure, Discourse, and Language. Edited by Adele Goldberg. Stanford: Center for the Study of Language and Information (CSLI), 113–130 [distributed by Cambridge University Press]Google Scholar logo with link to Google Scholar
. (2002). The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities. Basic Books.Google Scholar logo with link to Google Scholar
Hampe, Beate, Irene Mittelberg, Peter Uhrig, and Mark Turner. (2017). There-constructions ‘in the wild.’ Presentation at the 2017 International Conference on Multimodal Communication. 9–11 July. University of Osnabrück.
Hinnell, Jennifer. (2018). The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English. Cognitive Linguistics, 29(4), 773–806. Google Scholar logo with link to Google Scholar
Hoffmann, Thomas. (2017). From constructions to Construction Grammar. In: Barbara Dancygier, ed. The Cambridge Handbook of Cognitive Linguistics. Cambridge: Cambridge University Press, 284–309. Google Scholar logo with link to Google Scholar
Hoffmann, Thomas & Graeme Trousdale. (2013). The Oxford Handbook of Construction Grammar. Oxford University Press. Google Scholar logo with link to Google Scholar
Israel, Michael. (1996). The Way Constructions Grow. In Adele Goldberg, ed. Conceptual Structure, Discourse and Language. Stanford: CSLI, 217–230.Google Scholar logo with link to Google Scholar
Joo, Jungseock, Francis F. Steen & Song-Chun Zhu. (2015). Automated facial trait judgment and election outcome prediction: Social dimensions of face. Proceedings of the IEEE International Conference on Computer Vision, 3712–3720. Los Alamitos, CA: Institute of Electrical and Electronics Engineers. Google Scholar logo with link to Google Scholar
Kay, Paul, and Charles J. Fillmore. (1999). Grammatical constructions and linguistic generalizations: The What’s X doing Y? construction. Language 75: 1–33. Google Scholar logo with link to Google Scholar
Li, Weixin, Jungseock Joo, Hang Qi, & Song-Chun Zhu. (2017). Joint image-text news topic detection and tracking by multimodal topic and-or graph. IEEE Transactions on Multimedia 19(2). 367–381. Google Scholar logo with link to Google Scholar
Murphy, G.. (1996) On metaphoric representations. Cognition, 60, 173–204. Google Scholar logo with link to Google Scholar
Nesset, T., A. Endresen, L. Janda, A. Makarova, F. Steen & M. Turner. (2013). How ‘here and ‘now in Russian and English establish joint attention in TV news broadcasts. Russian Linguistics 37. 229–251. Google Scholar logo with link to Google Scholar
Pagán Cánovas, C. & J. Valenzuela. (2017). Timelines and multimodal constructions: Facing new challenges. Linguistic Vanguard 3(s1). Google Scholar logo with link to Google Scholar
Stamenković, Dušan. (2015). “The Effects of Animated Visual Stimuli on the Process of Conceptual Blending in Riddle Solving.” Facta Universitatis. Vol. 13, No 1, 2015, pp. 11–19. Available at SSRN: [URL]
Stec, Kashmiri & Eve Sweetser. (2013). Borobudur and Chartres: Religious spaces as performative real-space blends. In R. Caballero & J. E. Diaz Vera (Eds.), Sensuous cognition Berlin: Mouton De Gruyter. pp. 265–291. Google Scholar logo with link to Google Scholar
Steen, Francis F., Anders Hougaard, Jungseock Joo, Inés Olza, Cristóbal Pagán Cánovas, Anna Pleshakova, Soumya Ray, Peter Uhrig, Javier Valenzuela, Jacek Woźny and Mark Turner. (2018). Toward an Infrastructure for Data-driven Multimodal Communication Research. Linguistics Vanguard. Volume 4, Issue1, Pages 1–9. Google Scholar logo with link to Google Scholar
Steen, Francis and Mark Turner. (2013). “Multimodal Construction Grammar.” In Borkent, Michael, Barbara Dancygier, and Jennifer Hinnell, editors, Language and the Creative Mind. Stanford, CA: CSLI Publications. Pages 255–274.Google Scholar logo with link to Google Scholar
Turner, Mark. (1996). The Literary Mind: The Origins of Thought and Language. New York: Oxford University Press.Google Scholar logo with link to Google Scholar
. (2008). Frame Blending. In Frames, Corpora, and Knowledge Representation, edited by Rema Rossini Favretti. Bologna: Bononia University Press, 2008. 13–32.Google Scholar logo with link to Google Scholar
. (2014). The Origin of Ideas: Blending, Creativity, and the Human Spark. New York: Oxford University Press.Google Scholar logo with link to Google Scholar
. (2015). Blending in Language and Communication. In Ewa Dabrowska & Dagmar Divjak (eds.), Handbook of cognitive linguistics, Chapter 10. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
. (2017a). Multimodal form-meaning pairs for blended classic joint attention. Linguistics Vanguard 3(s1). Google Scholar logo with link to Google Scholar
. (2017b). Polytropos and communication in the wild. In Barbara Dancygier (ed.), The Cambridge handbook of cognitive linguistics, 93–98. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
. (2018). The Role of Creativity in Multimodal Construction Grammar. Zeitschrift für Anglistik und Amerikanistik. Volume 66, Issue 3. Pages 357–370. Google Scholar logo with link to Google Scholar
Turner, Mark, Maíra Avelar, & Milene Mendes de Oliveira. (2019). “Blended Classic Joint Attention and Multimodal Deixis.” Signo. 44:79, pages 03–09. Google Scholar logo with link to Google Scholar
Turner, Mark. (2020). “Constructions and Creativity.” In Hoffmann, Thomas, editor, Construction Grammar and creativity: Evolution, psychology, and cognitive science, a special issue of Cognitive Semiotics, 13:1. Google Scholar logo with link to Google Scholar
Zima, Elisabeth. (2014a). English multimodal motion constructions. A construction grammar perspective. Studies van de BKL – Travaux du CBL – Papers of the LSB, Volume 8. [URL]
. (2014b). Gibt es multimodale Konstruktionen? Eine Studie zu [V(motion) in circles] und [all the way from X PREP Y]. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 15. 1–48. [URL]
. (2017). Multimodal constructional resemblance. The case of English circular motion constructions. In Francisco Ruiz de Mendoza, Alba Luzondo & Paula Pérez-Sobrino (eds.), Constructing families of constructions. Human Cognitive Processing Series. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue