Cover not available

In:Time in Embodied Interaction: Synchronicity and sequentiality of multimodal resources
Edited by Arnulf Deppermann and Jürgen Streeck
[Pragmatics & Beyond New Series 293] 2018
► pp. 31–68

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 1
Forward-looking

Where do we go with multimodal projections?

Published online: 13 September 2018
DOI logo with link
https://doi.org/10.1075/pbns.293.01stu
Abstract
This chapter deals with the concepts of projection and conditional relevance by examining the temporal and sequential relationship between speech, gesture and gaze in deictic practices. It makes the following claims: Verbal deictics combined with embodied practices that direct the addressee’s attention to visible phenomena form a multimodal, intercorporeal gestalt. They constitute a request for the addressee’s gaze. This means that within the deictic referencing act, a multimodal summons-answer sequence unfolds that comprises recurrent gaze patterns both of the pointing and the gazing participant. These gaze practices are context-sensitive and temporally fine-tuned to the emerging verbal and kinetic practices, they occupy specific positions within the deictic act which in turn can be embedded within other actions such as instructions, requests etc.
Article outline
  • 1.Introduction
  • 2.Multimodal projection and conditional relevance across modalities and actions
  • 3.Multimodal projection within utterances and actions
  • 4.Projections between speech, gaze and gesture in deictic utterances
    • 4.1 Summoning the addressee’s gaze
    • 4.2 Monitoring the addressee’s gaze
    • 4.3 Repair after gaze monitoring
    • 4.4 Gaze in deictic summons-answer sequences: Evidence from dual mobile eye-tracking
  • 5. Conclusion
  • Acknowledgements
  • Notes
  • References
  • Appendix
References (64)
References
Auer, Peter. 2005. “Projection in Interaction and Projection in Grammar.” Text and Talk 25 (1): 7–36.Google Scholar logo with link to Google Scholar
. 2015. “The Temporality of Language in Interaction: Projection and Latency.” In Temporality in Interaction, ed. by Deppermann, Arnulf, and Susanne Günthner, 27–56. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Brooks, Rechele, and Andrew N. Meltzoff. 2014. “Gaze Following: A Mechanism for Building Social Connections between Infants and Adults.” In Mechanisms of Social Connection: From Brain to Group, ed. by Mikulincer, Mario, and Phillip R. Shave, 167–183. Washington, D.C.: American Psychological Association.Google Scholar logo with link to Google Scholar
Bühler, Karl. 2011[1934]. Theory of Language. The Representational Function of Language. Translated by D. F. Goodwin in collaboration with A. Eschbach. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Couper-Kuhlen, Elizabeth, and Dagmar Barth-Weingarten. 2011. “A system for transcribing talk-in-interaction: GAT 2 translated and adapted for English.” Gesprächsforschung – Onlinezeitschrift zur verbalen Interaktion 12, 1–51. Retrieved from [URL].Google Scholar logo with link to Google Scholar
Dausenschön-Gay, Ulrich, and Ulrich Krafft. 2009. “Preparing Next Actions in Routine Activities.” Discourse Processes 46: 247–268.Google Scholar logo with link to Google Scholar
Deppermann, Arnulf. 2015. “When Recipient Design Fails: Egocentric Turn-Design of Instructions in Driving Lessons Leading to Breakdowns of Intersubjectivity.” Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 16: 63–101.Google Scholar logo with link to Google Scholar
Deppermann, Arnulf, and Susanne Günthner (eds). 2015. Temporality in Interaction. Philadelphia: John Benjamins.Google Scholar logo with link to Google Scholar
Deppermann, Arnulf, and Reinhold Schmitt. 2007. “Koordination. zur Begründung eines neuen Forschungsgegenstandes.” In Koordination. Analysen zur multimodalen Interaktion, ed. by Schmitt, Reinhold, 15–54. Tübingen: Narr.Google Scholar logo with link to Google Scholar
De Stefani, Elwys, and Anne-Daniele Gazin. 2014. “Instructional Sequences in Driving Lessons: Mobile Participants and the Temporal and Sequential Organization of Actions.” Journal of Pragmatics 65: 63–79.Google Scholar logo with link to Google Scholar
Duchowski, Andrew T. 2003. Eye Tracking Methodology. Theory and Practice. Berlin: Springer.Google Scholar logo with link to Google Scholar
Fillmore, Charles. 1997. Lectures on Deixis. Stanford, CA: CSLI.Google Scholar logo with link to Google Scholar
Goffman, Erving. 1963. Behavior in Public Places. Notes on the Social Organization of Gatherings. New York: The Free Press.Google Scholar logo with link to Google Scholar
Goodwin, Charles. 1980. “Restarts, Pauses, and the Achievement of a State of Mutual Gaze at Turn-Beginning.” Sociological Inquiry 50 (3–4): 272–302.Google Scholar logo with link to Google Scholar
. 1994. “Professional Vision.” American Anthropologist 96 (3): 606–633.Google Scholar logo with link to Google Scholar
. 2000. “Action and Embodiment within Situated Human Interaction.” Journal of Pragmatics 32: 1489–1522.Google Scholar logo with link to Google Scholar
. 2003. “Pointing as Situated Practice.” In Pointing. Where Language, Culture, and Cognition Meet, ed. by Kita, Sotaro, 217–241. NJ: Erlbaum.Google Scholar logo with link to Google Scholar
. 2007. “Interactive Footing.” In Reporting Talk. Reported Speech in Interaction, ed. by Holt, Elizabeth, and Rebecca Clift, 16–46. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Goodwin, Charles, and Marjorie H. Goodwin. 1987. “Concurrent Operations on Talk. Notes on the Interactive Organization of Assessments.” IPrA Papers in Pragmatics 1 (1): 1–54.Google Scholar logo with link to Google Scholar
Goodwin, Marjorie H. 2006. “Participation, Affect, and Trajectory in Family Directive/Response Sequences.” Text and Talk 26: 513–541.Google Scholar logo with link to Google Scholar
Hausendorf, Heiko. 2003. “Deixis and Speech Situation Revisited. The Mechanism of Perceived Perception.” In Deictic Conceptualisation of Space, Time and Person, ed. by Lenz, Friedrich, 249–269. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Heath, Christian. 1986. Body Movement and Speech in Medical Interaction. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Keevallik, Leelo. 2013. “Here in Time and Space: Decomposing Movement in Dance Instruction.” In Interaction and Mobility. Language and the Body in Motion, ed. by Haddington, Pentti, Mondada, Lorenza, and Maurice Nevile, 345–370. Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
Kendon, Adam. 1990. Conducting Interaction. Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
. 2004. Gesture. Visible Action as Utterance. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Kita, Sotaro (ed). 2003. Pointing. Where Language, Culture, and Cognition Meet. Mahwah, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Land, Michael F. 2006. “Eye Movement and the Control of Actions in Everyday Life.” Progress in Retinal and Eye Research 25: 296–324.Google Scholar logo with link to Google Scholar
Levinson, Stephen C. 2013. “Action Ascription and Formation.” In The Handbook of Conversation Analysis, ed. by Sidnell, Jack, and Tanya Stivers, 103–130. Wiley: Blackwell.Google Scholar logo with link to Google Scholar
Lindwall, Oskar, and Anna Ekström. 2012. “Instruction-in-Interaction: The Teaching and Learning of a Manual Skill.” Human Studies 35: 27–49.Google Scholar logo with link to Google Scholar
Lindwall, Oskar, Lymer, Gustav, and Christian Greiffenhagen. 2015. “The Sequential Analysis of Instruction.” In The Handbook of Classroom Discourse and Interaction, ed. by Markee, Numa 142–157. Oxford: Wiley and Blackwell.Google Scholar logo with link to Google Scholar
Meyer, Christian, Streeck, Jürgen, and Scott Jordan (eds). (to appear). Intercorporeality. Emerging Socialities in Interaction. Oxford: Oxford University Press.
Mondada, Lorenza. 2009. “The Embodied and Negotiated Production of Assessments in Instructed Actions.” Research on Language and Social Interaction 42 (4): 329–361.Google Scholar logo with link to Google Scholar
. 2014. “The Local Constitution of Multimodal Resources for Social Interaction.” Journal of Pragmatics 65: 137–156.Google Scholar logo with link to Google Scholar
. 2015. “Multimodal Completions.” In Temporality in Interaction, ed. by Deppermann, Arnulf, and Susanne Günthner, 267–307. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Rauniomaa, Mira, and Tiina Keisanen. 2012. “Two multimodal formats for responding to requests.” Journal of Pragmatics 44, 829–842.Google Scholar logo with link to Google Scholar
Rossano, Federico. 2012. Gaze Behavior in Face-to-Face Interaction. Nijmegen: Max Planck Institute for Psycholinguistics Series.Google Scholar logo with link to Google Scholar
Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. 1974. “A Simplest Systematics for the Organization of Turn-Taking for Conversation.” Language 50(4), 696–735.Google Scholar logo with link to Google Scholar
Sacks, Harvey. 1972. “On the Analyzability of Stories by Children.” In Directions in Sociolinguistics. The Ethnography of Speaking, ed. by Gumperz, John J., and Dell Hymes, 329–345. New York: Holt, Rinehart and Winston.Google Scholar logo with link to Google Scholar
. 1984. “Notes on Methodology.” In Structures of Social Action. Studies in Conversation Analysis, ed. by Atkinson, John M., and John C. Heritage, 21–27. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Schegloff, Emanuel A. 1968. “Sequencing in Conversational Openings.” American Anthropologist 70 (6): 1075–1095.Google Scholar logo with link to Google Scholar
1992. “In Another Context.” In Rethinking Context. Language as an Interpretative Phenomenon, ed. by Goodwin, Charles, and Alessandro Duranti, 191–228. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
2007. Sequence Organization in Interaction. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Schegloff, Emanuel A., and Harvey Sacks. 1973. “Opening Up Closings.” Semiotica 8: 289–327.Google Scholar logo with link to Google Scholar
Selting, Margret et al. (2009). “Gesprächsanalytisches Transkriptionssystem 2 (GAT 2).” Gesprächsforschung – Onlinezeitschrift zur verbalen Interaktion 10, 353–402. Retrieved from [URL].Google Scholar logo with link to Google Scholar
Stivers, Tanya, and Federico Rossano. 2010. “A Scalar View of Response Relevance.” Research on Language and Social Interaction 43(1), 49–54.Google Scholar logo with link to Google Scholar
Streeck, Jürgen. 1988. “The Significance of Gesture: How it is Established.” IPrA Papers in Pragmatics 2(1-2), 60–83.Google Scholar logo with link to Google Scholar
. 1993. “Gesture as Communication I: Its Coordination with Gaze and Speech.” Communication Monographs 60 (4): 275–299.Google Scholar logo with link to Google Scholar
. 1995. “On Projection.” In Social Intelligence and Interaction. Expressions and Implications of Social Bias in Human Intelligence, ed. by Goody, Esther N., 87–110. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
. 2002. “Grammars, Words, and Embodied Meanings: On the Uses and Evolution of So and Like.” Journal of Communication 52 (3): 581–596.Google Scholar logo with link to Google Scholar
. 2007. “Geste und verstreichende Zeit: Innehalten und Bedeutungswandel der “bietenden Hand.” In Gespräch als Prozess. Linguistische Aspekte der Zeitlichkeit verbaler Interaktion, ed. by Hausendorf, Heiko, 157–180. Tübingen: Narr.Google Scholar logo with link to Google Scholar
Streeck, Jürgen, and Ulrike Hartge. 1992. “Previews: Gestures at the Transition Place.” In The Contextualization of Language, ed. by Auer, Peter, and Aldo Di Luzio, 135–158. Amsterdam: Benjamins.Google Scholar logo with link to Google Scholar
Stukenbrock, Anja. 2008. ““Wo ist der Hauptschmerz?” – Zeigen am eigenen Körper in der medizinischen Kommunikation.” Gesprächsforschung – Online-Zeitschrift zur verbalen Interaktion 9: 1–33.Google Scholar logo with link to Google Scholar
. 2009. “Referenz durch Zeigen: Zur Theorie der Deixis.” Deutsche Sprache 37: 289–315.Google Scholar logo with link to Google Scholar
. 2010. “Überlegungen zu einem multimodalen Verständnis der gesprochenen Sprache am Beispiel deiktischer Verwendungsweisen des Ausdrucks so.” InLiSt – Interaction and Linguistic Structures 47 ([URL]).
. 2014a. “Take the Words out of My Mouth: Verbal Instructions as Embodied Practices.” Journal of Pragmatics 65: 80–102.Google Scholar logo with link to Google Scholar
. 2014b. “Pointing to an ‘Empty’ Space: Deixis am Phantasma in Face-to-Face Interaction.” Journal of Pragmatics 74: 70–93.Google Scholar logo with link to Google Scholar
. 2015a. Deixis in der face-to-face-Interaktion. Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
. 2015b. “pi mal Daumen. Über eine kontextreiche Unschärferelation beim Zeigen.” In Sprache im Gebrauch: räumlich, zeitlich, interaktional, ed. by Bergmann, Pia, Birkner, Gilles, Peter, Spiekermann, Helmut, and Tobias Streck, 343–360. Heidelberg: Winter.Google Scholar logo with link to Google Scholar
. 2016. “Deiktische Praktiken: Zwischen Interaktion und Grammatik.” In Sprachliche und kommunikative Praktiken. Jahrburch 2015 des Instituts für Deutsche Sprache, ed. by Deppermann, Arnulf, Feilke, Helmut, and Angelika Linke, 81–126. Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
. 2017. “Intercorporeal Phantasms: Kinesthetic Alignment with Imagined Bodies in Self-Defense Trainings.” In: Intercorporeality. Emerging Socialities in Interaction, ed. by Meyer, Christian, Streeck, Jürgen, and Scott Jordan, 237–263. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
. in press. “Mobile Dual Eye-Tracking in Face-to-Face Interaction: The Case of Deixis and Joint Attention.” In Eye Tracking in Interaction. Studies on the Role of Eye Gaze in Dialogue, ed. by Brône, Geert, and Bent Oben. Amsterdam: John Benjamins.
Stukenbrock, Anja, and Karin Birkner. 2012. “Multimodale Ressourcen bei Stadtführungen.” In Deutschland als fremde Kultur: Vermittlungsverfahren in Touristenführungen, ed. by Costa, Marcella, and Bernd Müller-Jacquier, 214–243. München: Judicium.Google Scholar logo with link to Google Scholar
Svensson, Marcus S., Luff, Paul, and Christian Heath. 2009. “Embedding Instruction in Practice: Contingency and Collaboration during Surgical Training.” Sociology of Health and Illness 31 (6): 889–906.Google Scholar logo with link to Google Scholar
Cited by (26)

Cited by 26 other publications

Auer, Peter & Barbara Laner
2025. Laughter and gaze among talkers on a walk. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 208 ff. DOI logo
Stukenbrock, Anja & Angeliki Balantani
2025. When the establishment of joint attention becomes problematic. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 243 ff. DOI logo
Zima, Elisabeth, Peter Auer & Christoph Rühlemann
2025. Why research on gaze in social interaction needs mobile eye tracking. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 24 ff. DOI logo
Pan, Shiwen & Yunfeng Ge
2024. Demonstrative this/that and gestures. Pragmatics & Cognition 31:1 ► pp. 125 ff. DOI logo
Yagi, Junichi
2024. “Five” or “ten”: analysing a co-operative correction in Muay Thai coaching. Sports Coaching Review 13:1 ► pp. 107 ff. DOI logo
Avgustis, Iuliia & Florence Oloff
2023. Getting (Others) Involved with Smartphones: Participation in Showing Sequences in Multiparty Settings. In Complexity of Interaction, ► pp. 297 ff. DOI logo
Choe, Ann Tai & Junichi Yagi
2023. Embodied remembering in coordinated performances. Multimodal Communication 12:2 ► pp. 99 ff. DOI logo
Norrthon, Stefan
2023. Cueing in Theatre: Timing and Temporal Variance in Rehearsals of Scene Transitions. Human Studies 46:2 ► pp. 199 ff. DOI logo
Saalasti, Satu, Kati Pajo, Barbara Fox, Seija Pekkala & Minna Laakso
2023. Embodied-Visual Practices during Conversational Repair: Scoping Review. Research on Language and Social Interaction 56:4 ► pp. 311 ff. DOI logo
Schmidt, Axel & Arnulf Deppermann
2023. On the Emergence of Routines: An Interactional Micro-history of Rehearsing a Scene. Human Studies 46:2 ► pp. 273 ff. DOI logo
Krug, Maximilian
2022. Temporal procedures of mutual alignment and synchronization in collaborative meaning-making activities in a dance rehearsal. Frontiers in Communication 7 DOI logo
Krug, Maximilian
2025. Gaze aversion as a marker of disalignment in interactions. In Mobile Eye Tracking [Pragmatics & Beyond New Series, 351], ► pp. 165 ff. DOI logo
Deppermann, Arnulf & Alexandra Gubina
2021. When the Body Belies the Words: Embodied Agency With darf/kann ich? (“May/Can I?”) in German. Frontiers in Communication 6 DOI logo
Deppermann, Arnulf, Lorenza Mondada & Simona Pekarek Doehler
2021. Early Responses: An Introduction. Discourse Processes 58:4 ► pp. 293 ff. DOI logo
Deppermann, Arnulf & Axel Schmidt
2021. Micro-Sequential Coordination in Early Responses. Discourse Processes 58:4 ► pp. 372 ff. DOI logo
Ehmer, Oliver
2021. Synchronization in demonstrations. Multimodal practices for instructing body knowledge. Linguistics Vanguard 7:s4 DOI logo
Ehmer, Oliver
2025. Marking action accomplishment with non‑lexical vocalizations. Interactional Linguistics 5:1-2 ► pp. 22 ff. DOI logo
Ehmer, Oliver & Daniel Mandel
2021. Projecting action spaces. On the interactional relevance of cesural areas in co-enactments. Open Linguistics 7:1 ► pp. 638 ff. DOI logo
Li, Xiaoting
2021. Multimodal practices for negative assessments as delicate matters: Incomplete syntax, facial expressions, and head movements. Open Linguistics 7:1 ► pp. 549 ff. DOI logo
Salminen-Saari, Jessica F. A., Enrique Garcia Moreno-Esteva, Eeva Haataja, Miika Toivanen, Markku S. Hannula & Anu Laine
2021. Phases of collaborative mathematical problem solving and joint attention: a case study utilizing mobile gaze tracking. ZDM – Mathematics Education 53:4 ► pp. 771 ff. DOI logo
Singh, Ajit
2021. Situating embodied action plans: pre-enacting and planning actions within knowledge communication in sports training. Linguistics Vanguard 7:s4 DOI logo
Dimitrova, Nevena
2020. The Role of Common Ground on Object Use in Shaping the Function of Infants’ Social Gaze. Frontiers in Psychology 11 DOI logo
Stukenbrock, Anja
2020. Deixis, Meta-Perceptive Gaze Practices, and the Interactional Achievement of Joint Attention. Frontiers in Psychology 11 DOI logo
Stukenbrock, Anja & Anh Nhi Dao
2019. Joint Attention in Passing: What Dual Mobile Eye Tracking Reveals About Gaze in Coordinating Embodied Activities at a Market. In Embodied Activities in Face-to-face and Mediated Settings, ► pp. 177 ff. DOI logo
Tschacher, Wolfgang & Hermann Haken
2019. Outlook. In The Process of Psychotherapy, ► pp. 153 ff. DOI logo
Deppermann, Arnulf & Jürgen Streeck
2018. The body in interaction. In Time in Embodied Interaction [Pragmatics & Beyond New Series, 293], ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 28 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue