Cover not available

In:Mobile Eye Tracking: New avenues for the study of gaze in social interaction
Edited by Elisabeth Zima and Anja Stukenbrock
[Pragmatics & Beyond New Series 351] 2025
► pp. 24–66

Get fulltext
PDF logo with download linkDownload Chapter PDF
EPUB logo with download linkDownload Chapter EPUB

Chapter 2
Why research on gaze in social interaction needs mobile eye tracking

 | University of Freiburg
 | University of Freiburg
 | University of Freiburg
Open Access logo
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 13 May 2025
DOI logo with link
https://doi.org/10.1075/pbns.351.02zim
Abstract
This chapter challenges the prevailing practice in ethnomethodologically inspired interaction research (EMCA) of recording and analyzing gaze in social interactions from an observer’s perspective. Contrary to the assumption that this perspective is ‘natural’, we demonstrate systematic divergence between analysts’ and participants’ viewpoints and argue that ‘the standard procedure’ of video recording does not allow for a reliable reconstruction of when interactants look to or away from each other in a considerable number of cases. Three intercoder reliability studies, comparing the transcription of mutual gaze in triadic interactions from an observer’s perspective with eye tracking data, support this argument. They reveal the inherent limitations of gaze coding from an observer’s perspective, while showing that gaze transcription based on eye tracking data, which captures the participants’ perspective, is much less error-prone. It minimizes the need to infer gaze targets from ambiguous bodily cues and thus emerges as the preferred method for accurately reconstructing mutual gaze as part of interactional sense-making.
Article outline
  • 1.Introduction
  • 2.Epistemological and methodological questions of video recording in EMCA
  • 3.EMCA methodology and epistemology and the study of human gaze: Video recording versus eye tracking
    • Vis-à-vis
    • Side-by-side
    • L-shaped
    • Semi-circular
    • Triangular
    • Circular
    • Quandrangular
  • 4.Testing the reliability of gaze transcription in standard EMCA data versus eye tracking data
    • 4.1Study design
    • 4.2Results
      • Study 1a (no sound): Mutual gaze transcription from an observer’s perspective in muted clips
      • Study 1b (observer’s perspective, with sound)
      • Study 2: Transcribing (mutual) gaze on the basis of eye tracking data
  • 5.Conclusions
  • Notes
  • References
References (55)
References
Auer, Peter, Laner, Barbara, Pfeiffer, Martin, and Kerstin Botsch (2024). “Noticing and Assessing Nature: A Multimodal Investigation of the Format ‘Perception Imperative + Exclamative’ based on Mobile Eye-tracking Data.” In New Perspectives in Interactional Linguistic research, ed. by Margaret Selting, and Dagmar Barth-Weingarten, 245–277. Amsterdam: Benjamins. Google Scholar logo with link to Google Scholar
Bergmann, Jörg R. 1985. “Flüchtigkeit und methodische Fixierung sozialer Wirklichkeit: Aufzeichnungen als Daten der interpretativen Soziologie.“ In Entzauberte Wissenschaft: Zur Relativität und Geltung soziologischer Forschung, ed. by Wolfgang Bonß, and Heinz Hartmann, 299–320. Göttingen: Schwartz.Google Scholar logo with link to Google Scholar
Bowers, N. R., Boehm, A. E., Roorda, A. 2019. “The effects of fixational tremor on the retinal image”. Journal of Vision 19(11):8. Google Scholar logo with link to Google Scholar
Dalby Kristiansen, Elisabeth, and Gitte Rasmussen. 2021. “Eyetracking Recordings as Data in EMCA Studies: Exploring Possibilities and Limitations.” Social Interaction. Video-Based Studies of Human Sociality 4 (4). Google Scholar logo with link to Google Scholar
Deppermann, Arnulf, and Lorenza Mondada. 2018. “Overtaking as an Interactional Achievement: Video Analyses of Participants’ Practices in Traffic.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 19: 1–131. [URL]
Goffman, Erving. 1979. “Footing.” Semiotica 25: 1–29. Google Scholar logo with link to Google Scholar
Goodwin, Charles. 1981. Conversational Organization: Interaction between Speakers and Hearers. London: Academic Press.Google Scholar logo with link to Google Scholar
. 1993. “Recording Human Interaction in Natural Settings.” Pragmatics 3 (2): 181–209.Google Scholar logo with link to Google Scholar
. 1994. “Professional Vision.” American Anthropologist 96 (3): 606–633. Google Scholar logo with link to Google Scholar
Gubina, Alexandra. 2021. “Availability, Grammar, and Action Formation: On Simple and Modal Interrogative Request Formats in Spoken German.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 19: 1–131. [URL] 22, 272–303.
. 2022. Grammatik des Handelns in der sozialen Interaktion. Göttingen: Verlag für Gesprächsforschung.Google Scholar logo with link to Google Scholar
Heath, Christian, Hindmarsh, Jon, and Paul Luff (eds). 2010. Video in Qualitative Research. Analysing Social Interaction in Everyday Life. London: Sage. Google Scholar logo with link to Google Scholar
Hirschauer, Stefan. 2006. “Putting Things into Words. Ethnographic Description and the Silence of the Social.” Human Studies 29 (4): 413–441.
Holmqvist, Kenneth, Nyström, Marcus, Andersson, Richard, Dewhurst, Richard, Halszka, Jarodzka, and Joost van de Weijer (eds). 2011. Eye Tracking: A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Kendon, Adam. 1973. “The Role of Visible Behaviour in the Organization of Social Interaction.” In Social Communication and Movement: Studies of Interaction and Expression in Man and Chimpanzee, ed. by Mario von Cranach, and Ian Vine, 29–74. New York: Academic Press.Google Scholar logo with link to Google Scholar
. 1990. “Movement Coordination in Social Interaction: Some Examples Described.” In Conducting Interaction. Patterns of Behaviour in Focused Encounters, ed. by Adam Kendon, 91–115. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Knoblauch, Hubert. 2012. “Videography. Focused Ethnography and Video Analysis.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 69–84. Frankfurt am Main: Peter Lang.Google Scholar logo with link to Google Scholar
Rasmussen, Gitte and Kristiansen Dalby, Elisabeth. This volume. “The Influence of the Sopecificities of Gaze Behaviour on Emerging and Ensuing Interaction. A Contribution fo the Discussion of the Use of Eye-Tracking Recordings for EMCA Analysis.”
Landis, J. Richard, and Gary G. Koch. 1977. “The Measurement of Observer Agreement for Categorical Data.” Biometrics 33 (1): 159–174. Google Scholar logo with link to Google Scholar
Laurier, Eric. 2008. “Drinking up Endings: Conversational Resources of the Café.” Language and Communication 28 (2): 165–181. Google Scholar logo with link to Google Scholar
Laurier, Eric, and Chris Philo. 2006. “Natural Problems of Naturalistic Video Data.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 183–193. Oxford: Peter Lang.Google Scholar logo with link to Google Scholar
Laurier, Eric, and Tobias Boelt Back. 2023. “Recurrent Problems and Recent Experiments in Transcribing Video: Live Transcribing in Data Sessions and Depicting Perspective.” In Ethnomethodological Conversation Analysis in Motion: Emerging Methods and Technologies, ed. by Pentti Haddington, Tiina Eilittä, Antti Kamunen, Laura Kohonen-Aho, Tuire Oittinen, Lira Rautiainen, and Anna Vatanen, 245–263. London: Routledge. Google Scholar logo with link to Google Scholar
Luckmann, Thomas. 2012. “Some Remarks on Scores in Multimodal Sequential Analysis.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 29–34. Frankfurt am Main: Peter Lang.Google Scholar logo with link to Google Scholar
Lynch, Michael, and David Bogen. 1994. “Harvey Sacks’s Primitive Natural Science.” Theory, Culture & Society 11 (4): 65–110. Google Scholar logo with link to Google Scholar
McHugh, Mary L. 2012. “Interrater Reliability: The Kappa Statistic.” Biochem Med 22 (3): 276–82.Google Scholar logo with link to Google Scholar
McIlvenny, Paul. 2019. “Inhabiting Spatial Video and Audio Data: Towards a Scenographic Turn in the Analysis of Social Interaction.” Social Interaction. Video-Based Studies of Human Sociality 2 (1). Google Scholar logo with link to Google Scholar
Mohn, Elisabeth. 2012. “Permanent Work on Gazes. Video Ethnography as an Alternative Methodology.” In Video Analysis: Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 173–182. Frankfurt am Main: Peter Lang.Google Scholar logo with link to Google Scholar
Mondada, Lorenza. 2008. “Using Video for a Sequential and Multimodal Analysis of Social Interaction: Videotaping Institutional Telephone Calls.” Forum Qualitative Sozialforschung / Forum Qualitative Social Research 9 (3). Google Scholar logo with link to Google Scholar
. 2009. “Emergent Focused Interactions in Public Places: A Systematic Analysis of the Multimodal Achievement of a Common Interactional Space.” Journal of Pragmatics 41 (10): 1977–1997. Google Scholar logo with link to Google Scholar
. 2012a. “Video Recording as the Reflexive Preservation and Configuration of Phenomenal Features for Analysis.” In Video Analysis Methodology and Methods, ed. by Hubert Knoblauch, Bernt Schnettler, Jürgen Raab, and Hans-Georg Soeffner, 51–68. Frankfurt am Main: Peter Lang.Google Scholar logo with link to Google Scholar
. 2012b. “The Conversation Analytic Approach to Data Collection.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 32–56. Oxford: Wiley-Blackwell. Google Scholar logo with link to Google Scholar
. 2019. “Practices for Showing, Looking, and Videorecording: The Interactional Establishment of a Common Focus of Attention.” In Embodied Activities in Face-to-Face and Mediated Settings: Social Encounters in Time and Space, ed. by Elisabeth Reber, and Cornelia Gerhardt, 63–104. Cham: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Oloff, Florence. 2022. “The Particle jako (‘Like’) in Spoken Czech: From Expressing Comparison to Mobilizing Affiliative Responses.” Frontiers in Psychology 12 (662115). Google Scholar logo with link to Google Scholar
Pehkonen, Samu, Rauniomaa, Mirka, and Pauliina Siitonen. 2021. “Participating Researcher or Researching Participant? On Possible Positions of the Researcher in the Collection (and Analysis) of Mobile Video Data. Social Interaction. Video-Based Studies of Human Sociality 4 (2). Google Scholar logo with link to Google Scholar
Pekarek Doehler, Simona, Polak-Yitzhaki, Hilla, Li, Xiaoting, Stoenica, Ioana Maria, Havlík, Martin, and Leelo Keevallik. 2022. “Multimodal Assemblies for Prefacing a Dispreferred Response: A Cross-Linguistic Analysis.” Frontiers in Psychology 12 (689275). Google Scholar logo with link to Google Scholar
Peräkylä, Anssi, and Johanna E. Ruusuvuori. 2012. “Facial Expression and Interactional Regulation of Emotion.” In Emotion in Interaction, ed. by Anssi Peräkylä, and Marija L. Sorjonen, 64–91. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Rossano, Federico. 2013. “Gaze in Conversation.” In The Handbook of Conversation Analysis, ed. by Jack Sidnell, and Tanya Stivers, 308–329. Oxford: Wiley-Blackwell.Google Scholar logo with link to Google Scholar
Sacks, Harvey. [1967] 1992. Lectures on Conversation. (2 Volumes). Oxford: Basil Blackwell.Google Scholar logo with link to Google Scholar
Schmitz, H. Walter. 2020. “Wenn der Hörer sichtbar wird. Ch. Goodwins „The Interactive construction of a Sentence in Natural Conversation“ und die ethnomethodologische Konversationsanalyse.” In: Sine ira et studio: Disziplinübergreifende Annäherungen an die zwischenmenschliche Kommunikation, ed. by Robin Kurilla, Karin Kolb-Albers, Hannes Krämer, and Karola Pitsch, 97–115. Berlin: Springer. Google Scholar logo with link to Google Scholar
Schutz, Alfred. 1972. “The Problem of Social Reality.” In Collected Papers. Volume 1, ed. by Maurice Natanson. 277–293. The Hague: Martinus Nijhoff.Google Scholar logo with link to Google Scholar
Stivers, Tanya. 2021. “Is Conversation Built for Two? The Partitioning of Social Interaction.” Research on Language and Social Interaction 54 (1): 1–19. Google Scholar logo with link to Google Scholar
Stoenica, Ioana-Maria, and Sophia Fiedler. 2022. “Multimodal Practice for Mobilizing Response: The Case of Turn-final tu vois ‘You See’ in French Talk-in-Interaction.” Frontiers in Psychology 12: 249–268. Google Scholar logo with link to Google Scholar
Stukenbrock, Anja. 2015. Deixis in der face-to-face-Interaktion. Berlin, Boston: de Gruyter. Google Scholar logo with link to Google Scholar
. 2018. “Forward-Looking: Where Do We Go With Multimodal Projections?” In Modalities and Temporalities: Convergences and Divergences of Bodily Resources in Interaction, ed. by Arnulf Deppermann and Jürgen Streeck. 31–68. Amsterdam: Benjamins. Google Scholar logo with link to Google Scholar
. 2020. “Deixis, Meta-Perceptive Gaze Practices, and the Interactional Achievement of Joint Attention.” Frontiers in Psychology, 11. Google Scholar logo with link to Google Scholar
. 2021. “Multimodal Gestalts and Their Change Over Time: Is Routinization Also Grammaticalization?” Frontiers in Communication, 6 (6622406). Google Scholar logo with link to Google Scholar
Stukenbrock, Anja, and Anh-Nhi Dao. 2019. “Joint Attention in Passing: What Dual Mobile Eye Tracking Reveals About Gaze in Coordinating Embodied Activities at a Market.” In Embodied Activities in Face-to-face and Mediated Settings: Social Encounters in Time and Space, ed. by Elisabeth Reber, and Cornelia Gerhardt, 177–213. Cham: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Stukenbrock, Anja and Balantani, Angeliki. This volume. “When the Establishment of Joint Attention Becomes Problematic: How Participants Manage Divergent and Competing Foci of Attention.”
Tomasello, Michael. 2008. Origins of Human Communication. Cambridge: MIT Press. Google Scholar logo with link to Google Scholar
Tomasello, Michael, Hare, Brian, Lehmann, Hagen, and Josep Call. 2007. “Reliance on Head versus Eyes in the Gaze Following of Great Apes and Human Infants: The Cooperative Eye Hypothesis.” Journal of Human Evolution 52 (3): 314–320. Google Scholar logo with link to Google Scholar
Tuncer, Sylvaine, and Pentti Haddington. 2019. “Looking at and Seeing Objects: Instructed Vision and Collaboration in the Laboratory.” Gesprächsforschung — Online-Zeitschrift zur verbalen Interaktion 20: 435–360. [URL]
Wittenburg, Peter, Brugman, Hennie, Russel, Albert, Klassmann, Axel, and Han Sloetjes. 2006. “ELAN: a Professional Framework for Multimodality Research.” In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), 1556–1559. Genua: European Language Resources Association (ELRA).Google Scholar logo with link to Google Scholar
Zhisheng, Wang, Nagai, Yukari, Zhu, Dan, Liu, Jiahui, and Ninyu Zou. 2019. “Based on Creative Thinking to Museum Lighting Design Influences to Visitors Emotional Response Levels Theory Research.” IOP Conference Series: Materials Science and Engineering 573: 1–7. Google Scholar logo with link to Google Scholar
Zima, Elisabeth. 2020. “Gaze and Recipient Feedback in Triadic Storytelling Activities.” Discourse Processes 57 (9): 725–748. Google Scholar logo with link to Google Scholar
Cited by (3)

Cited by three other publications

Botsch, Kerstin, Peter Auer, Barbara Laner & Martin Pfeiffer
2025. Joint attention without language?. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 277 ff. DOI logo
Krug, Maximilian
2025. Gaze aversion as a marker of disalignment in interactions. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 165 ff. DOI logo
Zima, Elisabeth
2025. Multimodal Construction Grammar, DOI logo

This list is based on CrossRef data as of 14 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue