Cover not available

In:Multimodality and Cognitive Linguistics
Edited by María Jesús Pinar Sanz
[Benjamins Current Topics 78] 2015
► pp. 167–180

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Intermedial cognitive semiotics

Some examples of multimodal cueing in virtual environments

Published online: 15 October 2015
DOI logo with link
https://doi.org/10.1075/bct.78.11lop
In this chapter, intermediality is explored from an interdisciplinary perspective that uses neuroscience as well as cognitive-semiotic concerns and insights from online digital communication, presenting it as a process where biophysical, technological, and interpersonal factors interact. Shared attention as well as spatial and temporal cueing – eye contact and the sonic modality – are explored from a task-oriented and social interactive dimension. The spatiotemporal impact of the mediating context is highlighted by moving from the role of visual cueing, in a brief reference to Al Davison’s autobiographical graphic novel The spiral cage, to a more detailed analysis of Annie Abrahams’ (2010) online project A fragmented relation, where cueing is dependent not just on spatial frames but also on the temporal dynamics introduced by the aural dimension recorded in an online environment. The paper tangentially touches upon the role of affect in communication.
References (44)
Abrahams, A. (2010). A fragmented relation. Produced by <[URL]> CAC Centre d’Art Contemporain, Le Quartier, Quimper. <[URL]> <[URL]>.Google Scholar logo with link to Google Scholar
Bach, D., Schachinger, H., Neuhoff, J., Esposito F., & Di Salle, F. (2008). Rising sound intensity: An intrinsic warning cue activating the amygdala. Cerebral Cortex, 18, 145–150. Google Scholar logo with link to Google Scholar
Blake, R., Sobel, K.V., & James, T.W. (2004). Neural synergy between kinetic vision and touch. Psychological Science, 15(6), 397–402. Google Scholar logo with link to Google Scholar
Bouchardon, S., & LópezVarela, A. (2011). Making sense of the digital as embodied experience. CLCWeb: Comparative Literature and Culture, 13(3). Google Scholar logo with link to Google Scholar
Churchland, P. (2012). Braintrust: What neuroscience tells us about morality. Princeton: Princeton University Press.Google Scholar logo with link to Google Scholar
Critchley, M. (1977). Ecstatic and synaesthetic experiences during musical perception. In M. Critchley & R.A. Henson (Eds.). Music and the brain: Studies in the neurology of music (pp. 217–230). London: Heinemann.Google Scholar logo with link to Google Scholar
Davison, A. (2003). The spiral cage. London: Active Images.Google Scholar logo with link to Google Scholar
Ernst, M., Banks, M., & Bülthoff, H. (2000). Touch can change visual slant perception. Nature Neuroscience, 3(1), 69–73. Google Scholar logo with link to Google Scholar
Fauconnier, G. (1994). Mental spaces. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
. (1997). Mappings in thought and language. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
Fillmore, C. (1975). Santa Cruz lectures on deixis. Bloomington: Indiana University Linguistics Club.Google Scholar logo with link to Google Scholar
Forceville, C., & Urios-Aparisi, E. (2009). Multimodal metaphor. Berlin/New York: Mounton de Gruyter. Google Scholar logo with link to Google Scholar
Gallese, V., & Lakoff, G. (2005). The brain’s concepts: The role of the sensory-motor system in reason and language. Cognitive Neuropsychology, 22, 455–479. Google Scholar logo with link to Google Scholar
Gallese, V. (2006). Intentional attunement: A neurophysiological perspective on social cognition and its disruption in autism. Brain Res. Cog. Brain Res, 1079, 15–24.Google Scholar logo with link to Google Scholar
. (2008). Mirror neurons and the social nature of language: The neural exploitation hypothesis. Social Neuroscience, 3, 317–333. Google Scholar logo with link to Google Scholar
George, N., & Conty, L. (2008.) Facing the gaze of others. Clinical Neurophysiology, 38(3), 197–207. Google Scholar logo with link to Google Scholar
Glickstein, M., & Doron, K. (2008) Cerebellum: Connections and Functions. The Cerebellum, 7(4), 589–594. Google Scholar logo with link to Google Scholar
Gillespie, A., & Cornish, F. (2010). Intersubjectivity: Towards a dialogical analysis. Journal for the Theory of Social Behaviour, 40, 19–46. Google Scholar logo with link to Google Scholar
Grishakova, M. (2012). On cognitive and semiotic functions of shifters. Chinese Semiotic Studies, 8(2), 227–238. Google Scholar logo with link to Google Scholar
Guest, S., Catmur, C., Lloyd, D., & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161–171. Google Scholar logo with link to Google Scholar
Iacoboni, M. (2005). Understanding others: imitation, language, empathy. In S. Hurley & N. Chater (Eds.), Perspectives on imitation: From cognitive neuroscience to social science (pp. 77–99). Cambridge, MA: MIT Press.Google Scholar logo with link to Google Scholar
Jenson, D., & Iacoboni, M. (2011). Literary biomimesis: Mirror neurons and the ontological priority of representation. California Italian Studies, 2(1) <[URL]>.Google Scholar logo with link to Google Scholar
Kemmerer, D. (2006). The semantics of space: integrating linguistic typology and cognitive neuroscience. Neuropsychology, 44, 1607–1621. Google Scholar logo with link to Google Scholar
Kurylowicz, J. (1964). The inflectional categories of Indo-European. Heidelberg: Carl Winter Universitätsverlag.Google Scholar logo with link to Google Scholar
Lopez-Poveda, E., Palmer, A.R., & Meddis, R. (2010). The neurophysiological bases of auditory perception. New York: Springer Verlag. Google Scholar logo with link to Google Scholar
López-Varela, A. (2010). Exploring intercultural relations from the intersubjective perspectives offered through creative art in multimodal formats. Lexia, 5–6, 125–147.Google Scholar logo with link to Google Scholar
. (2011). Multimodal metaphor and intersubjective experiences: the importance of eye-contact in Davison’s graphic novel ‘The Spiral Cage’ and in Annie Abrahams Net-Project ‘On Collaboration’. In L. Masucci & G. Di Rosario (Eds.), Lavori del Convegno Palazzo degli Artista Italiani (pp. 307–324). Naples: Oficina di Letterature Electrónica.Google Scholar logo with link to Google Scholar
Maravita, A., Spence, Ch., Kennett, S., & Driver, J. (2002). Tool-use changes multimodal spatial interactions between vision and touch in normal humans. Cognition, 83, 25–34. Google Scholar logo with link to Google Scholar
Magosso, E., Serino, A., Pellegrino, G.D., & Ursino, M. (2010). Crossmodal links between vision and touch in spatial attention: A computational modeling study. Computational Intelligence and Neuroscience. [URL]Google Scholar logo with link to Google Scholar
Peirce, C.S. (1906). Prolegomena to an apology for pragmaticism, The Monist, 5(4), 492–546. Reprinted in C. Hartshorne & A. W. Weiss (Eds.), Collected papers of C. S. Peirce. Volume V(pp. 530–572). Cambridge, MA: Harvard University Press. Google Scholar logo with link to Google Scholar
. (1932). Division of signs. In C. Hartshorne & A.W. Weiss (Eds.), Collected papers of C. S. Peirce. Volume II. Cambridge, MA: Harvard University Press.Google Scholar logo with link to Google Scholar
. (1991). On the nature of signs. In J. Hoopes (Ed.), Peirce on signs. Chapel Hill: University of North Carolina Press.Google Scholar logo with link to Google Scholar
Pylyshyn, Z.W. (2000). Situating vision in the world. Trends in Cognitive Sciences, 4(5), 107–207. Google Scholar logo with link to Google Scholar
Rensink, R. (2000). The dynamic representation of scenes. Visual Cognition, 7, 17–42. Google Scholar logo with link to Google Scholar
Searle, J.R. (1995). The construction of social reality. London: Allen Lane.Google Scholar logo with link to Google Scholar
Schmidt, R.A. (1982). Motor control and learning: A behavioural emphasis. Champaign, IL: Human Kinetics Publishers Inc.Google Scholar logo with link to Google Scholar
. (1991). Motor learning and performance. Champaign, IL: Human Kinetics Publishers Inc.Google Scholar logo with link to Google Scholar
Tresilian, J. (2012). Sensorimotor control and learning: An introduction to the behavioral neuroscience of action. London: Palgrave Macmillan.Google Scholar logo with link to Google Scholar
Sperber, D., & Wilson, D. (1986). Relevance: Communication and cognition. Oxford: Blackwell.Google Scholar logo with link to Google Scholar
Stolorow, R.D., & Atwood, G.E. (1992). Contexts of being: The intersubjective foundations of psychological life. Hillsdale, NJ & London: The Analytic Press.Google Scholar logo with link to Google Scholar
Wade, N.J. (2010). Pioneers in eye movement. i perception, 1, 33–68. <[URL]>. Google Scholar logo with link to Google Scholar
Wind, J. (1989). Studies in language origins. Amsterdam/Philadelphia: John Benjamins. Google Scholar logo with link to Google Scholar
Zlatev, J. (2008). The co-evolution of intersubjectivity and bodily mimesis. In J. Zlatev, T. Racine, C. Sinha & E. Itkonen (Eds.), The shared mind: Perspectives on intersubjectivity (pp. 215–244). Amsterdam/Philadelphia: John Benjamins. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue