Cover not available

In:Eye-tracking in Interaction: Studies on the role of eye gaze in dialogue
Edited by Geert Brône and Bert Oben
[Advances in Interaction Studies 10] 2018
► pp. 265–300

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Chapter 11
Mobile dual eye-tracking in face-to-face interaction

The case of deixis and joint attention

Published online: 13 November 2018
DOI logo with link
https://doi.org/10.1075/ais.10.11stu
Abstract
In face-to-face interaction deixis, i.e. (the use of) a particular class of linguistic items that have grammaticalised the space-, time- and person-bound structure of the participants’ subjective orientation in the speech event (Bühler, 1965[1934]), is intricately connected to visible acts of demonstration (prototypically pointing) and joint attention. A growing body of publications within the field of conversation analysis and research on multimodality acknowledges the central role that pointing plays in acts of deictic reference (Eriksson, 2009; Fricke, 2007; Goodwin, 2003; Kendon, 2004; Kita, 2003; Mondada, 2012a; Stukenbrock, 2009, 2014a, 2014b, 2015). Surprisingly, eye gaze has remained an unexplored area although it serves a variety of crucial functions in the participants’ on-line organisation of a joint focus of attention on deictically foregrounded entities in the immediate spatial surroundings (Stukenbrock, 2009, 2010, 2014a, 2014b, 2015). The few existing studies mainly rely on video-recordings that do not allow a precise analysis of eye gaze.
Drawing on innovative mobile eye-tracking technology, my paper explores different forms of gaze behaviour that systematically occur when participants direct their interlocutor’s attention to visible entities in the surroundings by means of deictic pointing. My data consists of mobile eye-tracking recordings undertaken with two pairs of eye-tracking glasses worn by participants in non-laboratory, everyday settings ((1) shopping together at a market, (2) searching for a book in a library, (3) conducting an informal conversation). The analysis is based on frame-precisely synchronised split-screen videos consisting of two complementary eye-tracking videos which allow a moment-by-moment reconstruction of the way in which the participants coordinate talk, body movements and gaze in the emergent interaction.
Article outline
  • 1.Introduction
  • 2.Data, methodology and challenges
  • 3.Deictic practices in face-to-face interaction
    • 3.1What is deixis?
    • 3.2Verbal deixis combines with bodily pointing
    • 3.3Verbal deixis requests the addressee’s gaze
  • 4.Analyses: Gaze in deictic practices
    • 4.1The gesture-eye-link in demonstratio ad oculos
    • 4.2The target-eye-link in demonstratio ad oculos
    • 4.3The eye-to-eye-link in demonstratio ad oculos
  • 5.Conclusion: On eye-tracking natural gaze practices in demonstratio ad oculos
  • Conventions for the transcription gaze behaviour
    • The target-eye-link
    • The body-eye-link, more specifically the gesture-eye-link
    • The eye-to-eye-link
  • GAT 2 transcription conventions (GAT2, Selting et al. 2009; for the English translation cf. Couper-Kuhlen & Barth-Weingarten 2011)
    • Sequential structure
    • In- and outbreaths
    • Pauses
    • Segmental conventions
    • Laughter
    • Continuers
    • Accentuation
    • Final pitch movements of intonation phrases
    • Other conventions
  • Acknowledgments
  • Notes
  • References
References (77)
References
Argyle, M. & Cook, M. (1976). Gaze and Mutual Gaze. London: Cambridge University Press.Google Scholar logo with link to Google Scholar
Brooks, R. & Meltzoff, A. N. (2014). Gaze following: A mechanism for building social connections between infants and adults. In M. Mikulincer & P. R. Shaver (Eds.), Mechanisms of social connection: From brain to group (pp.167–183). Washington, D.C.: American Psychological Association.Google Scholar logo with link to Google Scholar
Bühler, K. (1965 [1934]). Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart: Gustav Fischer Verlag.Google Scholar logo with link to Google Scholar
(2011 [1934]). Theory of Language. The Representational Function of Language. (translated by D. F. Goodwin in collaboration with A. Eschbach). Amsterdam, Philadelphia: John Benjamins.Google Scholar logo with link to Google Scholar
Clark, A. T. & Gergle, D. (2011, September). Mobile Dual Eye-Tracking Methods: Challenges and Opportunities. Paper presented at DUET 2011: Dual Eye Tracking Workshop at ECSCW 2011, Aarhus, Denmark.Google Scholar logo with link to Google Scholar
(2012, February). Know what I’m talking about? Dual eye-tracking in multimodal reference resolution. Paper presented at DUET 2012: Dual Eye Tracking workshop at CSCW 2012, Bellevue, WA.Google Scholar logo with link to Google Scholar
Clark, H. H., Schreuder, R. & Buttrick, S. (1983). Common Ground and the Understanding of Demonstrative Reference. Journal of Verbal Learning and Verbal Behavior, 22(2), 245–258.Google Scholar logo with link to Google Scholar
Couper-Kuhlen, E. & Barth-Weingarten, D. (2011). A system for transcribing talk in interaction. GAT 2. English translation and adaption of Selting. Margret et al. (2009): Gesprächsanalytisches Transkriptionssystem 2. In: Gesprächsforschung - Online-Zeitschrift zur verbalen Interaktion, 12, 1–51.Google Scholar logo with link to Google Scholar
Couper-Kuhlen, E. & Selting, M. (2001). Studies in Interactional Linguistics. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Dao, A. N. (in prep.). Blickverhalten bei sprachlichen Lokalisierungen in der Interaktion. PhD thesis Freiburg/Br.
Deppermann, A. & Schmitt, R. (2007). Koordination. Zur Begründung eines neuen Forschungsgegenstandes. In R. Schmitt (Ed.), Koordination. Analysen zur multimodalen Interaktion (pp.15–54). Tübingen: Narr.Google Scholar logo with link to Google Scholar
De Stefani, E. (2010). Reference as an interactively and multimodally accomplished practice. Organizing spatial reorientation in guided tours. In M. Pettorino et al. (Eds.), Spoken Communication (pp.137–170). Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar logo with link to Google Scholar
De Stefani, E. & Gazin, A. -D. (2014). Instructional sequences in driving lessons: Mobile participants and the temporal and sequential organization of actions. Journal of Pragmatics, 65, 63–79.Google Scholar logo with link to Google Scholar
De Stefani, E. & Mondada, L. (2014). Reorganizing mobile formations: When ‘guided’ participants initiate reorientation in guided tours. Space and Culture, 17(2), 157–175.Google Scholar logo with link to Google Scholar
Diessel, H. (2006). Demonstratives, joint attention, and the emergence of grammar. Cognitive Linguistics, 17(4), 463–489.Google Scholar logo with link to Google Scholar
Duchowski, A. T. (2003). Eye Tracking Methodology. Theory and Practice. Berlin: Springer.Google Scholar logo with link to Google Scholar
Eriksson, M. (2009). Referring as Interaction: On the Interplay Between Linguistic and Bodily Practices. Journal of Pragmatics, 41, 240–262.Google Scholar logo with link to Google Scholar
Fillmore, C. (1997). Lectures on Deixis. Stanford, CA: CSLI Publ.Google Scholar logo with link to Google Scholar
Fricke, E. (2007). Origo, Geste und Raum. Lokaldeixis im Deutschen. Berlin: de Gruyter.Google Scholar logo with link to Google Scholar
Garfinkel, H. (1967). Studies in Ethnomethodology. Englewood Cliffs, Nj: Prentice-Hall.Google Scholar logo with link to Google Scholar
Gergle, D. & Clark, A. T. (2010). Effects of Shifting Spatial Context on Referential Form. Paper presented at the 20th Annual Meeting of the Society for Text and Discourse.Google Scholar logo with link to Google Scholar
(2011). See What I’m Saying? Using Dyadic Mobile Eye-Tracking to Study Collaborative Reference. Proceedings of CSCW (2011), 435–444.Google Scholar logo with link to Google Scholar
Goffman, E. (1963). Behavior in Public Places. Notes on the Social Organization of Gatherings. New York: The Free Press.Google Scholar logo with link to Google Scholar
(1981). Forms of Talk. Philadelphia: University of Pennsylvania Press.Google Scholar logo with link to Google Scholar
Goodwin, C. (1980). Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn-Beginning. Sociological Inquiry, 50(3–4), 272–302.Google Scholar logo with link to Google Scholar
(1981). Conversational Organization: Interaction Between Speakers and Hearers. New York: Academic Press.Google Scholar logo with link to Google Scholar
(2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32, 1489–1522.Google Scholar logo with link to Google Scholar
(2003). Pointing as Situated Practice. In S. Kita (Ed.), Pointing. Where Language, Culture, and Cognition Meet (pp.217–241). Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Gullberg, M. & Kita, S. (2009). Attention to Speech-Accompanying Gestures: Eye Movements and Information Uptake. Journal of Nonverbal Behavior, 33, 251–277.Google Scholar logo with link to Google Scholar
Haddington, P., Mondada, L. & Neville, M. (2013). Interaction and Mobility. Language and the Body in Motion. Berlin, Boston: de Gruyter.Google Scholar logo with link to Google Scholar
Hanks, W. F. (1990). Referential practice. Language and lived space among the Maya. Chicago: University of Chicago Press.Google Scholar logo with link to Google Scholar
Hanna, J. E. & Brennan, S. E. (2007). Speakers’ eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language, 57, 596–651.Google Scholar logo with link to Google Scholar
Hausendorf, H. (2003). Deixis and speech situation revisited. The mechanism of perceived perception. In F. Lenz (Ed.), Deictic Conceptualisation of Space, Time and Person (pp.249–269). Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Hepburn, A. & Bolden, G. (2012). The Conversation Analytic Approach to Transcription. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.57–76). Wiley: Blackwell.Google Scholar logo with link to Google Scholar
Hindmarsh, J. & Heath, C. (2000). Embodied reference: A study on deixis in workplace interaction. Journal of Pragmatics, 32(12), 1855–1878.Google Scholar logo with link to Google Scholar
Holler, J. & Kendrick, K. (2015). Unaddressed participants’ gaze in multi-person interaction: Optimizing recipiency. Frontiers in Psychology, 6(98), 1–14.Google Scholar logo with link to Google Scholar
Horsley, M., Eliot, M., Knight, B. A. & Reilly, R. (Eds.) (2014). Current Trends in Eye Tracking Research. Heidelberg: Springer.Google Scholar logo with link to Google Scholar
Keevallik, L. (2013). Here in time and space: Decomposing movement in dance instruction. In P. Haddington, L. Mondada & M. Nevile (Eds.), Interaction and Mobility. Language and the Body in Motion (pp.345–370). Berlin, Boston: de Gruyter.Google Scholar logo with link to Google Scholar
Kendon, A. (1967). Some function of gaze-direction in social interaction. Acta Psychologica, 26, 22–63.Google Scholar logo with link to Google Scholar
(1990). Conducting Interaction. Patterns of Behavior in Focused Encounters. Cambridge, New York: Cambridge University Press.Google Scholar logo with link to Google Scholar
(2004). Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Kita, S. (Ed.) (2003). Pointing. Where Language, Culture, and Cognition Meet. Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Land, M. F. (2006): Eye movement and the control of actions in everyday life. Progress in Retinal and Eye Research, 25, 296–324.Google Scholar logo with link to Google Scholar
Levinson, S. C. (1983). Pragmatics. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
(2004). Deixis. In L. R. Horn (Ed.), The handbook of pragmatics (pp.97–121). Oxford: Blackwell.Google Scholar logo with link to Google Scholar
(2012). Action Ascription and Formation. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.103–130). Wiley: Blackwell.Google Scholar logo with link to Google Scholar
Luhmann, N. (1984). Soziale Systeme. Grundriß einer allgemeinen Theorie. Frankfurt/M.: Suhrkamp.Google Scholar logo with link to Google Scholar
Mondada, L. (2007). Multimodal Resources for Turn-Taking: Pointing and the Emergence of Possible Next Speakers. Discourse Studies, 9(2), 195–226.Google Scholar logo with link to Google Scholar
(2012a). Deixis: an integrated interactional multimodal analysis. In P. Bergmann, J. Brenning, M. Pfeiffer & E. Reber (Eds.), Prosody and Embodiment in Interactional Grammar (pp.173–206). Berlin, Boston: de Gruyter.Google Scholar logo with link to Google Scholar
(2012b). The Conversation Analytic Approach to Data Collection. In J. Sidnell & T. Stivers (Eds.), The Handbook of Conversation Analysis (pp.32–56). Wiley: Blackwell.Google Scholar logo with link to Google Scholar
Pomerantz, A. (1984). Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp.57–101). Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
van Quine, W. O. (1960). Word and Object. Cambridge: Technology Press of the Massachusetts Institute of Technology.Google Scholar logo with link to Google Scholar
Rossano, F. (2012). Gaze Behavior in Face-to-Face Interaction. Nijmegen: Max Planck Institute for Psycholinguistics Series.Google Scholar logo with link to Google Scholar
Schegloff, E. A. (1968). Sequencing in Conversational Openings. American Anthropologist, 70(6), 1075–1095.Google Scholar logo with link to Google Scholar
(1996). Confirming Allusions: Toward an Empirical Account of Action. American Journal of Sociology, 102(1), 161–216.Google Scholar logo with link to Google Scholar
(2007). Sequence Organization in Interaction. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Schegloff, E. A. & Sacks, H. (1973). Opening Up Closings. Semiotica, 8, 289–327.Google Scholar logo with link to Google Scholar
Selting, M. et al. (2009). Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung – Onlinezeitschrift zur verbalen Interaktion, 10, 353–402.Google Scholar logo with link to Google Scholar
Sidnell, J. & Stivers, T. (Eds.) (2013). The Handbook of Conversation Analysis. Wiley: Blackwell.Google Scholar logo with link to Google Scholar
Streeck, J. (1993). Gesture as communication I: Its coordination with gaze and speech. Communication Monographs, 60, 275–299.Google Scholar logo with link to Google Scholar
(2002). Grammars, Words, and Embodied Meanings: On the Uses and Evolution of So and Like. Journal of Communication, 52(3), 581–596.Google Scholar logo with link to Google Scholar
(2009). Gesturecraft. The manu-facture of meaning. Amsterdam, Philadelphia: John Benjamins.Google Scholar logo with link to Google Scholar
(2014). Mutual gaze and recognition: Revisiting Kendon’s “Gaze direction in two-person conversation”. In M. Seyfeddinipur & M. Gullberg (Eds.), From Gesture in Conversation to Visible Action as Utterance. Essays in honor of Adam Kendon (pp.35–55). Amsterdam/Philadelphia: Benjamins.Google Scholar logo with link to Google Scholar
Stukenbrock, A. (2008). “Wo ist der Hauptschmerz?” – Zeigen am eigenen Körper in der medizinischen Kommunikation. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion, 9, 1–33.Google Scholar logo with link to Google Scholar
(2009). Referenz durch Zeigen: Zur Theorie der Deixis. Deutsche Sprache, 37, 289–315.Google Scholar logo with link to Google Scholar
(2010). Überlegungen zu einem multimodalen Verständnis der gesprochenen Sprache am Beispiel deiktischer Verwendungsweisen des Ausdrucks “so”. InLiSt –Interaction and Linguistic Structures, 47, 1–23.Google Scholar logo with link to Google Scholar
(2012). Imagined spaces as a resource in interaction. Bulletin Suisse de Linguistique Appliquée, 96, 141–161.Google Scholar logo with link to Google Scholar
(2014a). Take the words out of my mouth: Verbal instructions as embodied practices. Journal of Pragmatics, 65, 80–102.Google Scholar logo with link to Google Scholar
(2014b). Pointing to an ‘empty’ space: Deixis am Phantasma in face-to-face interaction. Journal of Pragmatics, 74, 70–93.Google Scholar logo with link to Google Scholar
(2015). Deixis in der face-to-face-Interaktion. Berlin, Boston: de Gruyter.Google Scholar logo with link to Google Scholar
(2016). Deiktische Praktiken: Zwischen Interaktion und Grammatik. In A. Deppermann, H. Feilke & A. Linke (Eds.), Sprachliche und kommunikative Praktiken (pp.81–126). Jahrburch 2015 des Instituts für Deutsche Sprache. Berlin, Boston: de Gruyter.Google Scholar logo with link to Google Scholar
(2018). Forward-looking: Where do we go with multimodal projections? In A. Deppermann & Jürgen Streeck (Eds.), Modalities and Temporalities: Convergences and divergences of bodily resources in interaction. Amsterdam: Benjamins, 31–68.Google Scholar logo with link to Google Scholar
Stukenbrock, A. & Dao, A. N. (accepted). Joint attention in passing. What dual mobile eye-tracking reveals about gaze in coordinating embodied activities on a market.
Stukenbrock, A. (in prep.). Divergent foci of visual attention in joint projects.
Tomasello, M. (2008). Origins of Human Communication. Cambridge/Massachusetts: MIT Press.Google Scholar logo with link to Google Scholar
Weiß, C. & Auer, P. (2016). Das Blickverhalten des Rezipienten bei Sprecherhäsitationen. Eine explorative Studie. Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion , 17, 132–167.Google Scholar logo with link to Google Scholar
Cited by (12)

Cited by 12 other publications

Huang, Shu-Yu
2025. Establishing joint attention to embodied actions with zheyang(zi) in Taiwan Mandarin cooking demonstrations. Chinese Language and Discourse. An International and Interdisciplinary Journal 16:2 ► pp. 271 ff. DOI logo
Krug, Maximilian
2025. Gaze aversion as a marker of disalignment in interactions. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 165 ff. DOI logo
Stukenbrock, Anja & Elisabeth Zima
2025. Introduction. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 1 ff. DOI logo
Neto Carvalho, Isabel, Mandy Schiefner-Rohs & Carina Troxler
2022. Eye-Viewing als Verfahren der erziehungswissenschaftlichen Medienforschung. Fokussierte Videographie mit Eye-Tracking-Technologie am Beispiel der Erforschung von medienbezogenen Praktiken in Schule und Unterricht. ZQF – Zeitschrift für Qualitative Forschung 23:2 ► pp. 262 ff. DOI logo
Rasmussen, Gitte & Elisabeth Dalby Kristiansen
2022. The sociality of minimizing involvement in self-service shops in Denmark: Customers’ multi-modal practices of being, getting, and staying out of the way. Discourse & Communication 16:2 ► pp. 200 ff. DOI logo
Rasmussen, Gitte & Elisabeth Dalby Kristiansen
2025. The influence of the specificities of gaze behavior on emerging and ensuing interaction. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 67 ff. DOI logo
Auer, Peter
2021. Turn-allocation and gaze: A multimodal revision of the “current-speaker-selects-next” rule of the turn-taking system of conversation analysis. Discourse Studies 23:2 ► pp. 117 ff. DOI logo
Auer, Peter
2021. Gaze selects the next speaker in answers to questions pronominally addressed to more than one co-participant. Interactional Linguistics 1:2 ► pp. 154 ff. DOI logo
Balantani, Angeliki
2021. Reference construction in interaction: The case of type-indicative “so”. Journal of Pragmatics 181 ► pp. 241 ff. DOI logo
Stukenbrock, Anja
2018. Forward-looking. In Time in Embodied Interaction [Pragmatics & Beyond New Series, 293], ► pp. 31 ff. DOI logo
Stukenbrock, Anja
2020. Deixis, Meta-Perceptive Gaze Practices, and the Interactional Achievement of Joint Attention. Frontiers in Psychology 11 DOI logo
Stukenbrock, Anja
2023. Temporality and the cooperative infrastructure of human communication: Noticings to delay and to accelerate onward movement in mobile interaction. Language & Communication 92 ► pp. 33 ff. DOI logo

This list is based on CrossRef data as of 14 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue