Cover not available

In:Visually Situated Language Comprehension
Edited by Pia Knoeferle, Pirita Pyykkönen-Klauck and Matthew W. Crocker
[Advances in Consciousness Research 93] 2016
► pp. 227–260

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Characterising visual context effects

Active, pervasive, but resource-limited

ORCiD logo with link
Published online: 10 March 2016
DOI logo with link
https://doi.org/10.1075/aicr.93.09kno
In this chapter, I will review recent research on visually situated language comprehension, and in doing so identify key characteristics of situated language comprehension. More specifically I will argue that both active visual context effects and the temporally coordinated interplay between visual attention and language comprehension are characteristic of situated comprehension, and are robust across a broad range of comprehension situations, spanning (a) different comprehension modalities (reading and spoken comprehension) and situations in which language is (versus isn’t) in accord with visual context; (b) different kinds of visual contexts (clipart depictions, photographs, and real-world objects and events); (c) speaker-based information such as eye-gaze and gestures; and (d) both concrete and abstract language. Because of their broad coverage ((a)–(d)), situated language comprehension paradigms are, in principle, well suited for developing a relatively comprehensive theory of situated language comprehension. One challenge in further specifying model predictions is the development of more detailed linking hypotheses between comprehension processes and one of the key measures used to examine situated comprehension (visual attention to objects across time).
References (93)
References
Abashidze, D., Carminati, M.N., & Knoeferle, P. (2014). How robust is the recent-event preference? In P. Bello, M. Guarini, M. McShane, & B. Scassellati (Eds.), Proceedings of the 36th Annual Meeting of the Cognitive Science Society (pp. 92-97). Cognitive Science Society.Google Scholar logo with link to Google Scholar
Abashidze, D., Knoeferle, P., Carminati, M.N., & Essig, K. (2011). The role of recent real-world versus future events in the comprehension of referentially ambiguous sentences: Evidence from eye tracking. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar logo with link to Google Scholar
Allopenna, P., Magnuson, J., & Tanenhaus, M. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory & Language, 38, 419-439. Google Scholar logo with link to Google Scholar
Altmann, G.T.M. (2004). Language-mediated eye-movements in the absence of a visual world: The ‘blank screen paradigm’. Cognition, 93, B79-B87. Google Scholar logo with link to Google Scholar
Altmann, G.T.M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264. Google Scholar logo with link to Google Scholar
. (2004). Now you see it, now you don’t: Mediating the mapping between language and the visual world. In J. Henderson & F. Ferreira (Eds.), The integration of language, vision and action (pp. 347-386). New York: Psychology Press.Google Scholar logo with link to Google Scholar
. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57, 502-518. Google Scholar logo with link to Google Scholar
. (2009). Discourse-mediation of the mapping between language and the visual world: Eye movements and mental representation. Cognition, 111, 55-71. Google Scholar logo with link to Google Scholar
Andersson, R., Ferreira, F., & Henderson, J.M. (2011). I see what you’re saying: The integration of complex speech and scenes during language comprehension. Acta Psychologica, 137, 208-216. Google Scholar logo with link to Google Scholar
Arai, M., van Gompel, R., & Scheepers, C. (2007). Priming ditransitive structures in comprehension. Cognitive Psychology, 54, 218-250. Google Scholar logo with link to Google Scholar
Arbib, M. (2005). From monkey-like action recognition to human language: An evolutionary framework for neurolinguistics. Behavioral and Brain Sciences, 28, 105-167. Google Scholar logo with link to Google Scholar
Barsalou, L.W. (1999). Perceptual and symbol systems. Behavioral and Brain Sciences, 22, 577-609.Google Scholar logo with link to Google Scholar
Brennan, S., Chen, X., Dickinson, C., Neider, M., & Zelinsky, G. (2007). Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition, 106, 1465-1477. Google Scholar logo with link to Google Scholar
Burigo, M., & Knoeferle, P. (2011). Visual attention during spatial language comprehension: Reference alone isn’t enough. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar logo with link to Google Scholar
Carminati, M.N., & Knoeferle, P. (2013). Effects of speaker emotional facial expression and listener age on incremental sentence processing. PLoS ONE, 8, e72559. Google Scholar logo with link to Google Scholar
Carstensen, L.L., Fung, H.H., & Charles, S.T. (2003). Socioemotional selectivity theory and the regulation of emotion in the second half of life. Motivation and Emotion, 27, 103-123. Google Scholar logo with link to Google Scholar
Casasanto, C. (2008). Similarity and proximity: When does close in space mean close in mind? Memory & Cognition, 36, 1047-1056. Google Scholar logo with link to Google Scholar
Castelhano, M., Mack, M., & Henderson, J. (2009). Viewing task influences eye movement control during active scene perception. Journal of Vision, 9(3), 1-15. Google Scholar logo with link to Google Scholar
Chambers, C.G., & San Juan, V. (2008). Perception and presupposition in real-time language comprehension: Insights from anticipatory processing. Cognition, 108, 26-50. Google Scholar logo with link to Google Scholar
Chambers, C.G., Tanenhaus, M.K., Filip, H., & Carlson, G.N. (2002). Circumscribing referential domains during real time language comprehension. Journal of Memory and Language, 47, 30-49. Google Scholar logo with link to Google Scholar
Chambers, C.G., Tanenhaus, M.K., & Magnuson, J.S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30, 687-696. Google Scholar logo with link to Google Scholar
Cook, S.W., & Tanenhaus, M.K. (2009). Embodied communication: Speakers’ gestures affect listeners’ actions. Cognition, 113, 98-104. Google Scholar logo with link to Google Scholar
Cooper, R. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107. Google Scholar logo with link to Google Scholar
Crocker, M.W., Knoeferle, P., & Mayberry, M. (2010). Situated sentence comprehension: The coordinated interplay account and a neurobehavioral model. Brain and Language, 112, 189-201. Google Scholar logo with link to Google Scholar
Dahan, D., & Tanenhaus, M. (2005). Looking at the rope when looking for the snake: Conceptually mediated eye movements during spoken-word recognition. Psychonomic Bulletin & Review, 12, 453-459. Google Scholar logo with link to Google Scholar
Duñabeitia, J.A., Aviles, A., Afonso, O., Scheepers, C., & Carreiras, M. (2008). Qualitative differences in the representation of abstract versus concrete words: Evidence from the visual-world paradigm. Cognition, 110, 284-292. Google Scholar logo with link to Google Scholar
Engelhardt, P.E., Ferreira, F., & Patsenko, E.G. (2010). Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology, 63, 639-664. Google Scholar logo with link to Google Scholar
Ferreira, F., Foucart, A., & Engelhardt, P.E. (2013). Language processing in the visual world: Effects of preview, visual complexity, and prediction. Journal of Memory and Language, 69, 165-182. Google Scholar logo with link to Google Scholar
Fodor, J. (1983). Modularity of mind. Cambridge, MA: MIT Press.Google Scholar logo with link to Google Scholar
Frazier, L., & Fodor, J.D. (1978). The sausage machine: A new two-stage parsing model. Cognition, 6, 291-325. Google Scholar logo with link to Google Scholar
Friederici, A.D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78-84. Google Scholar logo with link to Google Scholar
Ganis, G., Kutas, M., & Sereno, M.I. (1996). The search for ”common sense”: An electrophysiological study of the comprehension of words and pictures in reading. Journal of Cognitive Neuroscience, 8, 89-106. Google Scholar logo with link to Google Scholar
Goolkasian, P. (1996). Picture-word differences in a sentence verification task. Memory & Cognition, 24, 584-594. Google Scholar logo with link to Google Scholar
Griffin, Z., & Bock, K. (2000). What the eyes say about speaking. Psychological Science, 11, 274-279. Google Scholar logo with link to Google Scholar
Guerra, E., & Knoeferle, P. (2014). Effects of object distance on incremental semantic interpretation: Similarity is closeness. Cognition, 133, 535-552. Google Scholar logo with link to Google Scholar
Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P. (2011). The role of synchrony and ambiguity in speech-gesture integration during comprehension. Journal of Cognitive Neuroscience, 23, 1845-1854. Google Scholar logo with link to Google Scholar
Hagoort, P., Brown, C., & Groothusen, J. (1993). The syntactic positive shift (sps) as an ERP measure of syntactic processing. Language and Cognitive Processes, 8, 439-483. Google Scholar logo with link to Google Scholar
Hanna, J., & Brennan, S. (2007). Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596-615. Google Scholar logo with link to Google Scholar
Henderson, J.M., & Ferreira, F. (2004). Scene perception for psycholinguists. In J.M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world (pp. 1-58). New York: Psychology Press.Google Scholar logo with link to Google Scholar
Huang, Y.T., & Snedeker, J. (2009). Online interpretation of scalar quantifiers: Insight into the semantics –pragmatics interface. Cognitive Psychology, 58, 376-415. Google Scholar logo with link to Google Scholar
Huettig, F., & Altmann, G.T.M. (2005). Word meaning and the control of eye fixation: Semantic competitor effects and the visual world paradigm. Cognition, 96, 23-32. Google Scholar logo with link to Google Scholar
Huettig, F., Rommers, J., & Meyer, A.S. (2011). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica, 137, 151-171. Google Scholar logo with link to Google Scholar
Isaacowitz, D.M., Allard, E.S., Murphy, N.A., & Schlangel, M. (2009). The time course of age-related preferences toward positive and negative stimuli. The Journals of Gerontology Series B: Psychological Sciences and Social Sciences, 64B, 188-192. Google Scholar logo with link to Google Scholar
Kelly, S.D., Barr, D.J., Church, R.B., & Lynch, K. (1999). Offering a hand to pragmatic understanding: The role of speech and gesture in comprehension and memory. Journal of Memory and Language, 40, 577-592. Google Scholar logo with link to Google Scholar
Kelly, S.D., Creigh, P., & Bartolotti, J. (2010). Integrating speech and iconic gestures in a stroop-like task: Evidence for automatic processing. Journal of Cognitive Neuroscience, 22, 683-694. Google Scholar logo with link to Google Scholar
Kelly, S.D., Ozyurek, A., & Maris, E. (2010). Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science, 21, 260-267. Google Scholar logo with link to Google Scholar
Knoeferle, P. (2007). Comparing the time-course of processing initially ambiguous and unambiguous German SVO/OVS sentences in depicted events. In R. van Gompel, M. Fischer, W. Murray, & R. Hill (Eds.), Eye movement research: Insights into mind and brain (pp. 517-533). Oxford: Elsevier. Google Scholar logo with link to Google Scholar
. (in pressa). Language comprehension in rich non-linguistic contexts: Combining eye tracking and event-related brain potentials. In Towards a cognitive neuroscience of natural language use. Cambridge: Cambridge University Press.
. (in pressb). Visually situated language comprehension in children and in adults. In R.K. Mishra, N. Srinivasan, & F. Huettig (Eds.), Attention and vision in language processing. Springer Language and Cognition series.
Knoeferle, P., Carminati, M.N., Abashidze, D., & Essig, K. (2011). Preferential inspection of recent real-world events over future events: Evidence from eye tracking during spoken sentence comprehension. Frontiers in Psychology, 2, 376. Google Scholar logo with link to Google Scholar
Knoeferle, P., & Crocker, M.W. (2005). Incremental effects of mismatch during picture-sentence integration: Evidence from eye-tracking. In Proceedings of the 27th Annual Meeting of the Cognitive Science Conference (pp. 1166-1171). Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
. (2006). The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science, 30, 481-529. Google Scholar logo with link to Google Scholar
. (2007). The influence of recent scene events on spoken comprehension: Evidence from eye-movements. Journal of Memory and Language, 75, 519-543. Google Scholar logo with link to Google Scholar
Knoeferle, P., Crocker, M.W., Scheepers, C., & Pickering, M.J. (2005). The influence of the immediate visual context on incremental thematic role-assignment: Evidence from eye-movements in depicted events. Cognition, 95, 95-127. Google Scholar logo with link to Google Scholar
Knoeferle, P., Habets, B., Crocker, M.W., & Munte, T.F. (2008). Visual scenes trigger immediate syntactic reanalysis: Evidence from ERPs during situated spoken comprehension. Cerebral Cortex, 18, 789-795. Google Scholar logo with link to Google Scholar
Knoeferle, P., & Kreysa, H. (2012). Effects of speaker gaze on syntactic structuring. Frontiers in Psychology, 2, 376. Google Scholar logo with link to Google Scholar
Knoeferle, P., Urbach, T.P., & Kutas, M. (2011). Comprehending how visual context influences incremental sentence comprehension: Insights from ERPs and picture-sentence verification. Psychophysiology, 48, 495-506. Google Scholar logo with link to Google Scholar
Knoeferle, P., Urbach, T., & Kutas, M. (2014). Different mechanisms for role relations versus verb-action congruence effects: Evidence from ERPs in picture-sentence verification. Acta Psychologica, 152, 133-148. Google Scholar logo with link to Google Scholar
Kolk, H., Chwilla, D., Van Herten, M., & Oor, P. (2003). Structure and limited capacity in verbal working memory: A study with event-related potentials. Brain and Language, 85, 1-36. Google Scholar logo with link to Google Scholar
Kreysa, H. (2009). Coordinating speech-related eye movements between comprehension and production. Unpublished doctoral dissertation. University of Edinburgh, UK.Google Scholar logo with link to Google Scholar
Kreysa, H., & Knoeferle, P. (2011a). Effects of speaker gaze on spoken language comprehension: Task matters. In L. Carlson, C. Holscher, & T. Shipley (Eds.), Proceedings of the 33rd Annual Conference of the Cognitive Science Society. Cognitive Science Society.Google Scholar logo with link to Google Scholar
. (2011b). Peripheral speaker gaze facilitates spoken language comprehension: Syntactic structuring and thematic role assignment in German. In B. Kokinov, A. Karmiloff-Smith, & N.J. Nersessian (Eds.), Proceedings of the European Conference on Cognitive Science. New Bulgarian University Press.Google Scholar logo with link to Google Scholar
Kreysa, H., Knoeferle, P., & Nunnemann, E. (2014). Effects of speaker gaze versus depicted actions on visual attention during sentence comprehension. In M. McShane, B. Scassellati, P. Bello, & M. Guarini (Eds.), Proceedings of the 36th Annual Conference of the Cognitive Science Society (pp. 2513-2518). Cognitive Science Society.Google Scholar logo with link to Google Scholar
Kuperberg, G., Sitnikova, T., Caplan, D., & Holcomb, P. (2003). Electrophysiological distinctions in processing conceptual relationships within simple sentences. Cognitive Brain Research, 17, 117-129. Google Scholar logo with link to Google Scholar
Kutas, M., & Federmeier, K.D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event related brain potential (ERP). Annual Review of Psychology, 62, 621-647. Google Scholar logo with link to Google Scholar
Kutas, M., & Hillyard, S.A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207, 203-205. Google Scholar logo with link to Google Scholar
. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307, 161-163. Google Scholar logo with link to Google Scholar
Kutas, M., Van Petten, C., & Kluender, R. (2006). In M. Traxler & M. Gerns-bacher (Eds.), Handbook of psycholinguistics (2nd ed., pp. 659-724). New York: Elsevier. Google Scholar logo with link to Google Scholar
Mayberry, M., Crocker, M.W., & Knoeferle, P. (2009). Learning to attend: A connectionist model of situated language comprehension. Cognitive Science, 33, 449-496. Google Scholar logo with link to Google Scholar
Nappa, R., & Arnold, J. (2009). Paying attention to intention: Effects of intention (but not egocentric attention) on pronoun resolution. In Proceedings of the CUNY Conference (p. 262).
Nation, K., & Altmann, C.M.M.G.T.M. (2003). Investigating individual differences in childrens real-time sentence comprehension using language-mediated eye movements. Journal of Experimental Child Psychology, 86, 314-329. Google Scholar logo with link to Google Scholar
Osterhout, L., & Holcomb, P.J. (1992). Event-related brain potentials elicited by syntactic anomaly. Journal of Memory and Language, 31, 785-806. Google Scholar logo with link to Google Scholar
Pulvermuller, F., Harle, M., & Hummel, F. (2001). Walking or talking?: Behavioral and neurophysiological correlates of action verb processing. Brain and Language, 78, 143-168. Google Scholar logo with link to Google Scholar
Richardson, D., & Matlock, T. (2007). The integration of figurative language and static depictions: An eye movement study of fictive motion. Cognition, 102, 129-138. Google Scholar logo with link to Google Scholar
Richardson, D.C., & Dale, R. (2005). Looking to understand: The coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cognitive Science, 29, 1045-1060. Google Scholar logo with link to Google Scholar
Richardson, D.C., & Kirkham, N.Z. (2004). Multi-modal events and moving locations: Eye movements of adults and 6-month-olds reveal dynamic spatial indexing. Journal of Experimental Psychology: General, 133, 46-62. Google Scholar logo with link to Google Scholar
Salverda, A.P., Brown, M., & Tanenhaus, M.K. (2011). A goal-based perspective on eye movements in visual-world studies. Acta Psychologica, 137, 172-180. Google Scholar logo with link to Google Scholar
Scheepers, C., & Crocker, M.W. (2004). Constituent order priming from reading to listening: A visual-world study. In M. Carreiras & J.C. Clifton (Eds.), The on-line study of sentence comprehension: Eyetracking, ERP, and beyond. United Kingdom: Psychology Press.Google Scholar logo with link to Google Scholar
Sedivy, J.C., Tanenhaus, M.K., Chambers, C.G., & Carlson, G.N. (1999). Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-148. Google Scholar logo with link to Google Scholar
Spivey, M.J., & Geng, J.J. (2001). Oculomotor mechanisms activated by imagery and memory: Eye movements to absent objects. Psychological Research, 65, 235-241. Google Scholar logo with link to Google Scholar
Spivey, M.J., Tanenhaus, M.K., Eberhard, K.M., & Sedivy, J.C. (2002). Eye-movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481. Google Scholar logo with link to Google Scholar
Staudte, M., & Crocker, M. (2009). The effect of robot gaze on processing robot utterances. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the Cognitive Science Conference (pp. 431–436). Cognitive Science Society, Inc.Google Scholar logo with link to Google Scholar
Staudte, M., Crocker, M.W., Heloir, A., & Kipp, M. (2014). The influence of speaker gaze on listener comprehension: Contrasting visual versus intentional accounts. Cognition, 133, 317-328. Google Scholar logo with link to Google Scholar
Tanenhaus, M.K. (2004). That’s not it and "its" not "that": The role of conceptual composites in in-line reference resolution. In M. Carreiras & J. Charles Clifton (Eds.), On-line sentence processing: ERPs, eye movements, and beyond (pp. 209-228). Psychology Press.Google Scholar logo with link to Google Scholar
Tanenhaus, M.K., Magnuson, J.S., Dahan, D., & Chambers, C. (2000). Eye movements and lexical access in spoken- language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research, 29, 557-580. Google Scholar logo with link to Google Scholar
Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K., & Sedivy, J.C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632-634. Google Scholar logo with link to Google Scholar
Tatler, B., Wade, N., Kwan, H., Findlay, J., & Velichkovsky, B. (2010). Yarbus, eye movements, and vision. I-Perception, 1, 7-27. Google Scholar logo with link to Google Scholar
Underwood, G., Jebbett, L., & Roberts, K. (2004). Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. The Quarterly Journal of Experimental Psychology, 56, 165-182. Google Scholar logo with link to Google Scholar
Vissers, C., Kolk, H., van de Meerendonk, N., & Chwilla, D. (2008). Monitoring in language perception: Evidence from erps in a picture-sentence matching task. Neuropsychologia, 967-982. Google Scholar logo with link to Google Scholar
Wassenaar, M., & Hagoort, P. (2007). Thematic role assignment in patients with broca’s aphasia: Sentence-picture matching electrified. Neuropsychologia, 45, 716-740. Google Scholar logo with link to Google Scholar
Wu, Y.C., & Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology, 42, 654-667. Google Scholar logo with link to Google Scholar
Yarbus, A.L. (1967). Eye movements and vision. New York: Plenum Press. Google Scholar logo with link to Google Scholar
Zhang, L., & Knoeferle, P. (2012). Visual context effects on thematic role assignment in children versus adults: Evidence from eye tracking in German. In R.P. Cooper, N. Miyake, & D. Peebles (Eds.), Proceedings of the annual meeting of the cognitive science society (pp. 2593-2598). Boston, USA: The Cognitive Science Society.Google Scholar logo with link to Google Scholar
Cited by (1)

Cited by one other publication

Hernández-Gutiérrez, David, Francisco Muñoz, Jose Sánchez-García, Werner Sommer, Rasha Abdel Rahman, Pilar Casado, Laura Jiménez-Ortega, Javier Espuny, Sabela Fondevila & Manuel Martín-Loeches
2021. Situating language in a minimal social context: how seeing a picture of the speaker’s face affects language comprehension. Social Cognitive and Affective Neuroscience 16:5 ► pp. 502 ff. DOI logo

This list is based on CrossRef data as of 10 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue