In:Visually Situated Language Comprehension
Edited by Pia Knoeferle, Pirita Pyykkönen-Klauck and Matthew W. Crocker
[Advances in Consciousness Research 93] 2016
► pp. 323–356
Coordinating action and language
Published online: 10 March 2016
https://doi.org/10.1075/aicr.93.12far
https://doi.org/10.1075/aicr.93.12far
The overarching aim of this chapter is to highlight the strong interfacing between linguistic and motor systems, focusing especially on the degree to which manipulations of a visual context interact with linguistic manipulations in order to modulate the properties of manual movements elicited in response to a linguistic signal. First, we provide a review of recent work in the domain of embodied cognition that highlights the engagement of motor systems during language comprehension, including gestures, eye movements, and postural sway. In the context of this body of work, we then focus on the notion that continuous and non-ballistic computer-mouse movement trajectories can serve as a viable motoric index of various aspects of language processing. The results of studies that have recorded mouse-movement trajectories in visual-world displays will be reviewed, with the goal of highlighting the methodological benefits of data contained in the recorded trajectories, along with their theoretical consequences. New data will be presented from a study in which both eye- and hand-movements were simultaneously recorded, speaking to the relationship between the two movement-related dependent measures. Finally, we will highlight the degree to which mouse-movements around a visual display can be informative for domains other than syntactic processing, focusing on new data from research designed to investigate the processing of verbal aspect.
References (115)
Allopenna, P.D., Magnuson, J.S., & Tanenhaus, M.K. (1998). Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language, 38, 419-439.
Altmann, G.T.M., Garnham, A., & Dennis, Y. (1992). Avoiding the garden-path: Eye movements in context. Journal of Memory and Language, 31, 685-712.
Altmann, G.T.M., Garnham, A., & Henstra, J. (1994). Effects of syntax in human sentence parsing: Evidence against a structure-based proposal mechanism. Journal of Experimental Psychology: Learning, Memory, and Cognition, 20, 209-216.
Altmann, G.T.M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition, 73, 247-264.
Altmann G.T.M., & Steedman, M. (1988). Interaction with context during human sentence processing. Cognition, 30, 191-238.
Altmann, G.T.M., van Nice, K., Garnham, A., & Henstra, J.A. (1998). Late closure in context. Journal of Memory and Language. 38, 459-484.
Anderson, S.E., Farmer, T.A., Goldstein, M., Schwade, J., & Spivey, M. (2011). Individual differences in measures of linguistic experience account for variability in the sentence processing skill of five-year-olds. In I. Arnon & E.V. Clark (Eds.), Experience, variation, and generalization: Learning a first language (pp. 203-221). Amsterdam, The Netherlands: John Benjamins.
Anderson, S.E., Matlock, T., Fausey, C., & Spivey, M.J. (2008). On the path to understanding on-line processing of grammatical aspect. In
Proceedings of the 30th Annual Conference of the Cognitive Science Society
(pp. 143-148). Mahwah, NJ: Lawrence Erlbaum Associates.
Anderson, S.E., Matlock, T., & Spivey, M. (2013) Grammatical aspect and temporal distance in motion descriptions. Frontiers in Psychology, 4.
Bever, T.G. (1970). The cognitive basis for linguistic structures. In J.R. Hayes (Ed.), Cognition and the growth of cognition (pp. 279-362). New York, NY: Wiley.
Boulenger, V., Roy, A.C., Paulignan, Y., Deprez, V., Jeannerod, M., & Nazir, T.A. (2006). Cross-talk between language processes and overt motor behavior in the first 200 msec of processing. Journal of Cognitive Neuroscience, 18, 1607-1615.
Britt, M.A. (1994). The interaction of referential ambiguity and argument structure in the parsing of prepositional phrases. Journal of Memory and Language, 33, 251-283.
Brown, C.M., Van Berkum, J.J.A., & Hagoort, P. (2000). Discourse before gender: An event-related brain potential study on the interplay of semantic and syntactic information during spoken language understanding. Journal of Psycholinguistic Research, 29, 53-68.
Calvert, S., Rideout, V., Woolard, J., Barr, R., & Strouse, G. (2005). Age, ethnicity, and socioeconomic patterns in early computer use: A national survey. American Behavioral Scientist 48, 590-607.
Casasanto, D., & Lozano (2007). Embodied language production: Evidence from gesture, speech, disfluency, and motor action. Paper presented at
Embodied Sentence Processing Conference: Behavioral, Neuropsychological, and Computational Perspectives
. Saerbruecken, Germany.
Cooper, R.M. (1974). The control of eye fixation by the meaning of spoken language: A new methodology for the real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6, 84-107.
Cisek, P., & Kalaska, J.F. (2005). Neural correlates of reaching decisions in dorsal premotor cortex: Specification of multiple direction choices and final selection of action. Neuron, 45, 801-814.
Dale, R., & Duran, N.D. (2011). The cognitive dynamics of negated sentence verification. Cognitive Science, 1-14.
Dale, R., Kehoe, C., & Spivey, M.J. (2007). Graded motor responses in the time course of categorizing exemplars. Memory and Cognition, 35, 15-28.
Dale, R., Roche, J.M., Snyder, K., & McCall, R. (2008). Exploring action dynamics as an index of paired-associate learning. PLoS ONE, 3, e1728.
Doyle, M., & Walker, R. (2001). Curved saccade trajectories: Voluntary and reflexive saccades curve away from irrelevant distractors. Experimental Brain Research, 139, 333-344.
Duran, N.D., Dale, R., & McNamara, D. (2010). The action dynamics of overcoming the truth. Psychonomic Bulletin & Review, 17, 486-491.
Engelhardt, P.E., Bailey, K.G., & Ferreira, F. (2006). Do speakers and listeners observe the Gricean Maxim of Quantity? Journal of Memory and Language, 54, 554-573.
Farmer, T.A., Anderson, S., & Spivey, M.J. (2007). Gradiency and visual context in syntactic garden-paths. Journal of Memory and Language, 57, 570-595.
Farmer, T.A., Cargill, S.E., Hindy, N., Dale, R., & Spivey, M.J. (2007). Tracking the continuity of language comprehension: Computer-mouse trajectories suggest parallel syntactic processing. Cognitive Science, 31, 889-909.
Farmer, T.A., Christiansen, M.H., & Kemtes, K.A. (2005). Sentence processing in context: The impact of experience on individual differences. In
Proceedings of the 27th Annual Meeting of the Cognitive Science Society
(pp. 642-647). Mahwah, NJ: Lawrence Erlbaum.
Farmer, T.A., Davidow, J., & McCandliss, B.D. (2009). Integrating eye- and hand-movements to more accurately index competition-based effects in the visual worldparadigm. Poster presented the
22nd Annual Meeting of the CUNY Conference on Human Sentence Processing
. Davis, CA.
Farmer, T.A., Liu, R., Metha, N., & Zevin, J. (2009). Native language experience influences perceived similarity of second language vowel categories. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 2588-2593). Austin, TA: Cognitive Science Society.
Ferraina, S., Johnson, P.B., Garasto, M.R., Battaglia-Mayer, A., Ercolani, L., Bianchi, L., Lacquaniti, F., et al. (1997). Combination of hand and gaze signals during reaching: activity in parietal area 7m of the monkey. Journal of Neurophysiology, 77, 1034-1038.
Ferreira, F., Foucart, A., & Engelhardt, P.E. (2013). Language processing in the visual world: Effects of preview, visual complexity, and prediction. Journal of Memory and Language, 69, 165-182.
Ferreira, F., & Clifton, C. (1986). The independence of syntactic processing. Journal of Memory and Language, 25, 248-368.
Ferretti, T.R., Kutas, M., & McRae, K. (2007). Verb aspect and the activation of event knowledge in semantic memory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 33, 182-196.
Frazier, L., & Fodor, J. (1978). The sausage machine: A new two-stage parsing model. Cognition, 6, 291-325.
Freeman, J.B., & Ambady, N. (2009). Motions of the hand expose the partial and parallel activation of stereotypes. Psychological Science, 20, 1183-1188.
. (2010). MouseTracker: Software for studying real-time mental processing using a computer mouse-tracking method. Behavior Research Methods, 42, 226-241.
Freeman, J.B., Ambady, N., Rule, N.O., & Johnson, K.L. (2008). Will a category cue attract you? Motor output reveals dynamic competition across person construal. Journal of Experimental Psychology: General, 137, 673-690.
Freeman, J.B., Dale, R., & Farmer, T.A. (2011). Hand in motion reveals mind in motion. Frontiers in Psychology, 2, article 59.
Gaskell, M.G., & Marslen-Wilson, W. (1999). Ambiguity, competition, and blending in spoken word recognition. Cognitive Science, 23, 439-462.
Glenberg, A.M. & Kashak, M.P. (2002). Grounding language in action. Psychonomics Bulletin and Review, 9, 558-565.
Gold, J., & Shadlen, M. (2000). Representation of a perceptual decision in developing oculomotor commands. Nature, 404, 390-394.
Gold, J.I., & Shadlen, M.N. (2001). Neural computations that underlie decisions about sensory stimuli. Trends in Cognitive Sciences, 5, 10-16.
Goodale, M., Pélisson, D., & Prablanc, C. (1986). Large adjustments in visually guided reaching do not depend on vision of the hand or perception of target displacement. Nature, 320, 48-750.
Godijn, R., & Theeuwes, J. (2002). Programming of endogenous and exogenous saccades: Evidence for a competitive integration model. Journal of Experimental Psychology: Human Perception and Performance, 28, 1039-1054.
Green, M., & Mitchell, D. (2006). Absence of real evidence against competition during syntactic ambiguity resolution. Journal of Memory and Language, 55, 1-17.
Hauk, O., Johnsrude, I., & Pulvermuller, F. (2004). Somatotopic representation of action words in the motor and premotor cortex. Neuron, 41, 301-307.
Hauk, O., & Pulvermuller, F. (2004). Neurophysiological distinction of action words in fronto-central cortex. Human Brain Mappings, 21, 191-201.
Havas, D.A., Glenberg, A.M., Gutowski, K.A., Lucarelli, M.J., & Davidson, R.J. (2010). Cosmetic use of botulinum toxin-A affects processing of emotional language. Psychological Science, 21, 895-900.
Henis, E.A., & Flash, T. (1995). Mechanisms underlying the generation of averaged modified trajectories. Biological Cybernetics, 72, 407-419.
Hindy, N.C., Hamilton, R., Houghtling, A.S., Coslett, H.B., & Thompson-Schill, S.L. (2009). Computer mouse-tracking reveals TMS disruptions of prefrontal function during semantic retrieval. Journal of Neurophysiology, 102, 3405-3413.
Huette, S., Kello, C.T., Rhodes, T., & Spivey, M.J. (2013). Drawing from memory: Hand-eye coordination at multiple scales. PloS one, 8(3), e58464.
Johnson, P.B., Ferraina, S., Bianchi, L., & Caminiti, R. (1996). Cortical networks for visual reaching: Physiological and anatomical organization of frontal and parietal lobe arm regions. Cerebral Cortex, 6, 102-119.
Kalaska, J.F., Scott, S., Cisek, P., & Sergio, L.E. (1997). Cortical control of reaching movements. Current Opinion in Neurobiology, 7, 849-859.
Knoeferle, P., & Crocker, M.W. (2006). The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science, 30, 481-529.
Langacker, R.W. (1987). Foundations of cognitive grammar: Theoretical prerequisites. Stanford, CA: Stanford University Press.
Levy, J., & Staub, A. (2012). Concurrent eye-tracking and mouse-tracking in the visual world paradigm. Poster presented at the
53rd Annual Meeting of the Psychonomic Society
. Minneapolis, MN.
Louwerse, M.M. (2008). Embodied relations are encoded in language. Psychonomic Bulletin & Review, 15, 838-844.
Luce, P., Goldinger, S., Auer, E., & Vitevitch, M. (2000). Phonetic priming, neighborhood activation, and PARSYN. Perception and Psychophysics, 62, 615-625.
Madden, C.J., & Zwann, R.A. (2003). How does verb aspect constrain event representations? Memory & Cognition, 31, 663-672.
Magliano, J.P., & Schleich, M.C. (2000). Verb aspect and situation models. Discourse Processes, 29, 83-112.
MacDonald, M., Pearlmutter, N., & Seidenberg, M. (1994). The lexical nature of syntactic ambiguity resolution. Psychological Review, 101, 676-703.
Mahon, B.Z., & Caramazza, A. (2008). A critical look at the embodied cognition hypothesis and a new proposal for grounding conceptual content. Journal of Physiology – Paris, 102, 59-70.
Magnuson, J.S. (2005). Moving hand reveals dynamics of thought.
Proceedings of the National Academy of Sciences
, 102, 9995-9996.
Markman, A., & Brendl, C. (2005). Constraining theories of embodied cognition. Psychological Science, 16, 6-10.
Matlock, T., Fausey, C., Cargill, S., & Spivey, M. (2007). On the path toward understanding the dynamics of aspect descriptions in motion events. Paper presented at
48th Annual Meeting of the Psychonomic Society
. Long Beach, CA.
McClelland, J.L., & Elman, J.L. (1986). Interactive processes in speech perception: The TRACE model. In D.E. Rumelhart & J.L. McClelland (Eds.), Parallel distributed processing (Vol. II, pp. 58-121). Cambridge, UK: Cambridge University Press.
McKinstry, C., Dale, R., & Spivey, M.J. (2008). Action dynamics reveal parallel competition in decision making. Psychological Science, 19, 22-24.
McMurray, B., Tanenhaus, M.K., & Aslin, R.N. (2002). Gradient effects of within-category phonetic variation on lexical access. Cognition, 86, B33-B42.
McRae, K., Spivey-Knowlton, M.J., & Tanenhaus, M.K. (1998). Modeling the effects of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 37, 283-312.
Morrow, D.G. (1985). Prominent characters and events organize narrative understanding. Journal of Memory and Language, 24, 304-319.
Nazir, T.A., Boulenger, V., Roy, A., Silber, B., Jeannerod, M., & Paulignan, Y. (2007). Language-induced motor perturbations during the execution of a reaching movement. Journal of Cognitive Neuroscience, 18, 1607-1615.
Port, N.L., & Wurtz, R.H. (2003). Sequential activity of simultaneously recorded neurons in the superior colliculus during curved saccades. Journal of Neurophysiology, 90, 1887-19003.
Pulvermüller, F., Hauk, O., Nikulin, V.V., & Ilmoniemi, R.J. (2005). Functional links between motor and language systems. European Journal of Neuroscience, 21, 793-797.
Pylyshyn, Z. (1984). Computation and cognition: Toward a foundation for cognitive science. Cambridge, Mass: MIT Press.
Rosenbaum, D.A. (2005). The Cinderella of psychology: The neglect of motor control in the science of mental life and behavior. The American Psychologist, 60, 308-317.
Schmidt, T. (2002). The finger in flight: Real-time motor control by visually masked color stimuli. Psychological Science, 13, 112-118.
Sedivy, J.C., K Tanenhaus, M., Chambers, C.G., & Carlson, G.N. (1999). Achieving incremental semantic interpretation through contextual representation. Cognition, 71, 109-147.
Sheliga, B.M., Riggio, L., & Rizzolatti, G. (1994). Orienting of attention and eye movements. Experimental Brain Research, 98, 507-522.
Shen, L., & Alexander, G.E. (1997). Neural correlates of a spatial sensory-to-motor transformation in primary motor cortex. Journal of Neurophysiology, 77, 1171-1194.
Smith, J.R., Treat, T.A., McMurray, B., & Farmer, T.A. (2014). Clothing style and attractiveness influence online processing of women’s sexual interest. Poster presented at the
26th Annual Convention of the Association for Psychological Science
. San Francisco, CA.
. (in preparation). Men’s online processing of women’s sexual interest: Generalization of dynamic models of decision making.
Snedeker, J., & Trueswell, J. (2004). The developing constraints on parsing decisions: The role of lexical-biases and referential scenes in child and adults sentence processing. Cognitive Psychology, 49, 238-299.
Song, J.H., & Nakayama, K. (2006). Role of focal attention on latencies and trajectories of visually guided manual pointing. Journal of Vision, 6, 982-99.
. (2009). Hidden cognitive states revealed in choice reaching tasks. Trends in Cognitive Sciences, 13, 360-366.
Spivey, M.J., Anderson, S.E., & Farmer, T.A. (2013). Putting syntax in context. In R.P.G. van Gompel (Ed.), Sentence processing (pp. 115-135). Psychology Press.
Spivey, M.J., Dale, R., Knoblich, G., & Grosjean, M. (2010). Do curved reaching movements emerge from competing perceptions? Journal of Experimental Psychology: Human Perception and Performance, 36, 251-254.
Spivey, M., Grosjean, M., & Knoblich, G. (2005). Continuous attraction toward phonological competitors. Proceedings of the National Academy of Sciences, 102, 10393-10398.
Spivey, M.J., & Tanenhaus, M.K. (1998). Syntactic ambiguity resolution in discourse: Modeling the effects of referential context and lexical frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition, 24, 1521-1543.
Spivey, M., Tanenhaus, M.K., Eberhard, K. & Sedivy, J. (2002). Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution. Cognitive Psychology, 45, 447-481.
Spivey-Knowlton, M.J., & Sedivy, J. (1995). Resolving attachment ambiguities with multiple constraints. Cognition, 55, 227-267.
Tanenhaus, M.K. (2007). Spoken language comprehension: Insights from eye movements. In G. Gaskell (Ed.), Oxford handbook of psycholinguistics (pp. 309-326). Oxford: Oxford University Press.
Tanenhaus, M.K., Spivey-Knowlton, M., Eberhard, K., & Sedivy, J. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632-1634.
Theeuwes, J., Olivers, C.N.L., & Chizk, C.L. (2005). Remembering a location makes the eyes curve away. Psychological Science, 16, 196-199.
Thothathiri, M., & Snedeker, J. (2008). Syntactic priming during language comprehension in three- and four-year-old children. Journal of Memory and Language, 58, 188-213.
Traxler, M., Pickering, M.J., & Clifton, C. (1998). Adjunct attachment is not a form of lexical ambiguity resolution. Journal of Memory and Language, 39, 558-592.
Trueswell, J.C., Sekerina, I., Hill, N.M., & Logrip, M.L. (1999). The kindergarten path effect: Studying on-line sentence processing in young children. Cognition, 73, 89-134.
Trueswell, J.C., Tanenhaus, M.K., & Garnsey, S.M. (1994). Semantic influences on parsing: Use of thematic role information in syntactic disambiguation. Journal of Memory and Language, 33, 285-318.
van Berkum, J.J.A., Brown, C.M., & Hagoort, P. (1999a). Early referential context effects in sentence processing: Evidence from event –related potentials. Journal of Memory and Language, 41, 147-182.
. (1999b). Semantic integration in sentences and discourse: Evidence from the N400. Journal of Cognitive Neuroscience, 11, 657-671.
van Berkum, J.J.A., Brown, C.M., Hagoort, P., & Zwitserlood, P. (2003). Event-related brain potentials reflect discourse-referential ambiguity in spoken-language comprehension. Psychophysiology, 40, 235-248.
van der Wel, R.P.R.D., Eder, J., Mitchel, A., Walsh, M., & Rosenbaum, D. (2009). Trajectories emerging from discrete versus continuous processing models in phonological competitor tasks. Journal of Experimental Psychology: Human Perception and Performance, 32, 588-594.
van Gompel, R.P.G., Pickering, M.J., Pearson, J., & Liversedge, S.P. (2005). Evidence against competition during syntactic ambiguity resolution. Journal of Memory and Language, 52, 284-307.
van Gompel, R., Pickering, M.J., & Traxler, M. (2001). Reanalysis in sentence processing: Evidence against current constraint-based and two-stage models. Journal of Memory and Language, 52, 284-307.
Wifall, T., Buss, A.T., Farmer, T.A., Spencer, J.P., & Hazeltine, E. (submitted). Reaching into response selection: Stimulus and response similarity influence central operations.
Cited by (1)
Cited by one other publication
This list is based on CrossRef data as of 10 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
