Cover not available

In:Dimensions of Iconicity
Edited by Angelika Zirker, Matthias Bauer, Olga Fischer and Christina Ljungberg
[Iconicity in Language and Literature 15] 2017
► pp. 167–190

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Bridging the gap between image and metaphor through cross-modal iconicity

An interdisciplinary model

 | Linnaeus University
Published online: 8 September 2017
DOI logo with link
https://doi.org/10.1075/ill.15.10ell
Our minds are capable of perceiving similarities not only within the same but also across different sensory areas and different cognitive domains. Iconicity is representation based on similarity, and cross-modal iconicity, which is an extremely widespread phenomenon, should be understood as iconicity that crosses the borders of different kinds of material, spatiotemporal, and sensorial modes, and, furthermore, the border between sensory structures and cognitive configurations. For instance, a visual entity may resemble and thus iconically represent something that is auditory or abstractly cognitive. The aim of this semiotic study, substantially based on empirical findings in psychological, cognitive, and neurological research, is to suggest a general theoretical framework for conceptualizing cross-modal iconicity and relating different kinds of mono-modal and cross-modal iconicity to each other. Its chief argument is that perception and conception of images and metaphors should be understood as the two extremes in a continuum of iconic representation where cross-modal iconicity bridges the apparent gap between mono-modal, sensory-based iconicity and cognitive iconicity. It offers an outline of material, spatiotemporal, and sensorial modes and their interrelations; a thorough account of cross-modal iconicity; a conceptual structure for charting degrees of similarity and iconicity with the aid of cross-modality; an array of examples illustrating the continuum of iconicity from image to metaphor; and a brief discussion of the notion of image schema as an explanatory factor for cross-modal iconicity.
Article outline
  • 1.Introduction
  • 2.Material, spatiotemporal, and sensorial modes
  • 3.Cross-modal iconicity
  • 4.Degrees of similarity and iconicity
  • 5.From image to metaphor
  • 6.Image schemas
  • 7.Conclusion
  • References
References (50)
References
Ahlner, F. & Zlatev, J. 2010. Cross-modal iconicity: A cognitive semiotic approach to sound symbolism. Sign Systems Studies 38: 298–348. Google Scholar logo with link to Google Scholar
Anderson, E. R. 1998. A Grammar of Iconism. Madison NJ: Farleigh Dickinson University Press.Google Scholar logo with link to Google Scholar
Bankieris, K. & Simner, J. 2015. What is the link between synaesthesia and sound symbolism? Cognition 136: 186–195. Google Scholar logo with link to Google Scholar
Bauer, M. & Ernst, C. 2010. Diagrammatik: Einführung in ein kultur- und medienwissenschaftliches Forschungsfeld. Bielefeld: Transcript. Google Scholar logo with link to Google Scholar
Benovsky, J. 2012. Photographic representation and depiction of temporal extension. Inquiry 55(2): 194–213. Google Scholar logo with link to Google Scholar
Bremner, A. J., Caparos, S., Davidoff, J., Fockert, J. de, Linnell, K. J. & Spence, C. 2013. “Bouba” and “Kiki” in Namibia? A remote culture make similar shape–sound matches, but different shape–taste matches to Westerners. Cognition 126: 165–172. Google Scholar logo with link to Google Scholar
Brochard, R., Tassin, M. & Zagar, D. 2013. Got rhythm … for better and for worse: Cross-modal effects of auditory rhythm on visual word recognition. Cognition 127: 214–219. Google Scholar logo with link to Google Scholar
Cacciari, C. 2008. Crossing the senses in metaphorical language. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 425–443. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Cienki, A. 2005. Image schemas and gesture. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 421–441. Berlin: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
Colapietro, V. 2011. Image, diagram, and metaphor: Unmined resource and unresolved questions. In Semblance and Signification [Iconicity in Language & Literarure 10], P. Michelucci, O. Fischer & C. Ljungberg (eds), 157–171. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Cowles, J. T. 1935. An experimental study of the pairing of certain auditory and visual stimuli. Journal of Experimental Psychology 18(4): 461–469. Google Scholar logo with link to Google Scholar
Eco, U. 1976. A Theory of Semiotics. Bloomington IN: Indiana University Press. Google Scholar logo with link to Google Scholar
Elleström, L. 2010a. The modalities of media: A model for understanding intermedial relations. In Media Borders, Multimodality and Intermediality, L. Elleström (ed.), 11–48. Houndmills: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
. 2010b. Iconicity as meaning miming meaning, and meaning miming form. In Signergy [Iconicity in Language and Literature 9], J. Conradie, R. Johl, M. Beukes, O. Fischer & C. Ljungberg (eds), 73–100. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
. 2013. Spatiotemporal aspects of iconicity. In Iconic Investigations [Iconicity in Language and Literature 12], L. Elleström, O. Fischer & C. Ljungberg (eds), 95–117. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
. 2014. Material and mental representation: Peirce adapted to the study of media and arts. The American Journal of Semiotics 30(1–2): 83–138. Google Scholar logo with link to Google Scholar
. Forthcoming. A medium-centered model of communication. Semiotica.
Forceville, C. 2009. Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research. In Multimodal Metaphor, C. Forceville & E. Urios-Aparisi (eds), 19–42. Berlin: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
Fryer, L., Freeman, J. & Pring, L. 2014. Touching words is not enough: How visual experience influences haptic–auditory associations in the “Bouba–Kiki” effect. Cognition 132: 164–173. Google Scholar logo with link to Google Scholar
Goswami, U. 2001. Analogical reasoning in children. In The Analogical Mind: Perspectives from Cognitive Science, D. Gentner, K. J. Holyoak & B. N. Kokinov (eds), 437–470. Cambridge MA: The MIT Press.Google Scholar logo with link to Google Scholar
Johnson, M. 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. Chicago IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
Karwoski, T. F., Odbert, H. S. & Osgood, C. E. 1942. Studies in synesthetic thinking, II: The role of form in visual responses to music. Journal of General Psychology 26: 199–222. Google Scholar logo with link to Google Scholar
Köhler, W. 1929. Gestalt Psychology. New York NY: Horace Liveright.Google Scholar logo with link to Google Scholar
Kress, G. & van Leeuwen, T. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Hodder Arnold.Google Scholar logo with link to Google Scholar
Martino, G. & Marks, L. E. 2000. Cross-modal interaction between vision and touch: The role of synesthetic correspondence. Perception 29: 745–754. Google Scholar logo with link to Google Scholar
Morris, C. 1971. Writings on the General Theory of Signs. The Hague: Mouton. Google Scholar logo with link to Google Scholar
Mossbridge, J. A., Grabowecky, M. & Suzuki, S. 2011. Changes in auditory frequency guide visual–spatial attention. Cognition 121: 133–139. Google Scholar logo with link to Google Scholar
Nöth, W. 2001. Semiotic foundations of iconicity in language and literature. In The Motivated Sign [Iconicity in Language and Literature 2], O. Fischer & M. Nänny (eds), 17–28. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Odbert, H. S., Karwoski, T. F. & Eckerson, A. B. 1942. Studies in synesthetic thinking: I. Musical and verbal associations of color and mood. Journal of General Psychology 26: 153–173. Google Scholar logo with link to Google Scholar
Peirce, C. S. 1932. Collected Papers of Charles Sanders Peirce, II: Elements of Logic, C. Hartshorne & P. Weiss (eds). Cambridge MA: Harvard University Press.Google Scholar logo with link to Google Scholar
Perlman, M., Tanner, J. E. & King, B. J. 2012. A mother gorilla’s variable use of touch to guide her infant: Insights into iconicity and the relationship between gesture and action. In Developments in Primate Gesture Research [Gesture Studies 6], S. Pika & K. Liebal (eds), 55–72. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Perniss, P., Thompson, R. L. & Vigliocco, G. 2010. Iconicity as a general property of language: Evidence from spoken and signed languages. Frontiers in Psychology 1: 227. Google Scholar logo with link to Google Scholar
Perniss, P. & Vigliocco, G. 2014. The bridge of iconicity: From a world of experience to the experience of language. Philosophical Transactions of the Royal Society: Biological Sciences 369. Google Scholar logo with link to Google Scholar
Perry, L. K., Perlman, M. & Lupyan, G. 2015. Iconicity in English and Spanish and its relation to lexical category and age of acquisition. PLoS One 10(9). Google Scholar logo with link to Google Scholar
Popova, Y. 2005. Image schemas and verbal synaesthesia. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 395–419. Berlin: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
Ramachandran, V. S. & Hubbard, E. M. 2005. The emergence of the human mind: Some clues from synesthesia. In Synesthesia: Perspectives From Cognitive Neuroscience, N. Sagiv & L. C. Robertson (eds), 147–190. Oxford: OUP.Google Scholar logo with link to Google Scholar
Röder, B., Pagel, B. & Heed, T. 2013. The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing. Cognition 126: 301–306. Google Scholar logo with link to Google Scholar
Rohrer, T. 2005. Image schemata in the brain. In From Perception to Meaning: Image Schemas in Cognitive Linguistics, B. Hampe (ed.), 165–196. Berlin: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
Smith J. D., Flemming, T. M., Boomer, J., Beran, M. J. & Church, B. A. 2013. Fading perceptual resemblance: A path for rhesus macaques (Macaca mulatta) to conceptual matching? Cognition 129: 598–614. Google Scholar logo with link to Google Scholar
Smith, L. B. & Heise, D. 1992. Perceptual similarity and conceptual structure. In Percepts, Concepts and Categories, B. Burns (ed.), 233–272. Amsterdam: Elsevier.Google Scholar logo with link to Google Scholar
Spence, C. 2011. Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4): 971–995. Google Scholar logo with link to Google Scholar
Stein, B. E., Spence, C. & Calvert, G. 2004. The Handbook of Multisensory Processes. Cambridge MA: The MIT Press.Google Scholar logo with link to Google Scholar
Sutherland, C. A. M., Thut, G. & Romei, V. 2014. Hearing brighter: Changing in-depth visual perception through looming sounds. Cognition 132: 312–323. Google Scholar logo with link to Google Scholar
Sweeny, T. D., Guzman-Martinez, E., Ortega, L., Grabowecky, M. & Suzuki, S. 2012. Sounds exaggerate visual shape. Cognition 124: 194–200. Google Scholar logo with link to Google Scholar
Tabakowska, E. 2003. Iconicity and literary translation. In From Sign to Signing [Iconicity in Language and Literature 3], W. G. Müller & O. Fischer (eds), 361–376. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
. 2005. Iconicity as a function of point of view. In Outside-In – Inside-Out [Iconicity in Language and Literature 4], C. Maeder, O. Fischer & W. J. Herlofsky (eds), 375–387. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Usnadze, D. 1924. Ein experimenteller Beitrag zum Problem der psychologischen Grundlagen der Namengebung. Psychologische Forschung 5(1): 24–43. Google Scholar logo with link to Google Scholar
Yildirim, I. & Jacobs, R. A. 2013. Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies. Cognition 126: 135–148. Google Scholar logo with link to Google Scholar
Zbikowski, L. M. 2008. Metaphor and music. In The Cambridge Handbook of Metaphor and Thought, R. W. Gibbs (ed.), 502–524. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Ziembowicz, M., Nowak, A. & Winkielman, P. 2013. When sounds look right and images sound correct: Cross-modal coherence enhances claims of pattern presence. Cognition 129: 273–278. Google Scholar logo with link to Google Scholar
Cited by (12)

Cited by 12 other publications

Hou, Guanhua & Yongsheng Pan
2025. What’s Behind the Graphic? The Impact of Pictogram-Semantic Mapping Structure on Icon Understanding. International Journal of Human–Computer Interaction 41:20 ► pp. 12959 ff. DOI logo
Bonifazi, Anna
2024. Cross-modal iconicity in songs about weeping. Semiotica 2024:256 ► pp. 1 ff. DOI logo
Holm, Anne
2024. Sensing the Past: Multimodal Adjectives. In The Stylistics of Embodiment, ► pp. 163 ff. DOI logo
Holm, Anne
2024. Perceptual Metaphors as Cognitive Resources. In The Stylistics of Embodiment, ► pp. 191 ff. DOI logo
Alarauhio, Juha-Pekka, Tiina Räisänen, Jarkko Toikkanen & Riikka Tumelius
2022. Introduction: Multimodality and Intermediality in the North. In Shaping the North Through Multimodal and Intermedial Interaction [Arctic Encounters, ], ► pp. 1 ff. DOI logo
Elleström, Lars
2022. Symbolicity, language, and mediality. Semiotica 2022:247 ► pp. 1 ff. DOI logo
Salmose, Niklas
2020. Sensorial Aesthetics: Cross-Modal Stylistics in Modernist Fiction. Text Matters: A Journal of Literature, Theory and Culture :10 ► pp. 321 ff. DOI logo
Bateman, John A
2019. Transmediality and the End of Disembodied Semiotics. International Journal of Semiotics and Visual Rhetoric 3:2 ► pp. 1 ff. DOI logo
Elleström, Lars & Filip Cieślak
2019. Identifying, Construing, and Bridging over Media Borders. Tekstualia 3:58 ► pp. 21 ff. DOI logo
Fröhlich, Marlen, Christine Sievers, Simon W. Townsend, Thibaud Gruber & Carel P. van Schaik
2019. Multimodal communication and language origins: integrating gestures and vocalizations. Biological Reviews 94:5 ► pp. 1809 ff. DOI logo
Lugea, Jane
2018. The year’s work in stylistics 2017. Language and Literature: International Journal of Stylistics 27:4 ► pp. 329 ff. DOI logo
Pethő, Ágnes
2018. A Good Concept Should Be both Very Concrete and Very Abstract.”. Acta Universitatis Sapientiae, Film and Media Studies 15:1 ► pp. 191 ff. DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue