Cover not available

Article published In: Interaction Studies
Vol. 21:2 (2020) ► pp.268–292

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

“Talk to you later”

Doing social robotics with conversation analysis. Towards the development of an automatic system for the prediction of disengagement

 | Télécom Paris | Institut polytechnique de Paris | I3 CNRS
 | Télécom Paris | Institut polytechnique de Paris | Université Paris-Saclay
Published online: 20 May 2020
DOI logo with link
https://doi.org/10.1075/is.19001.roll
Abstract
This article presents an applied discussion of the possibility of integrating conversation analysis (CA) methodology into that of machine learning. The aim is to improve the detection of that which resembles disengagement in the interaction between a robot and a human. We offer a novel analytical assemblage at the heart of the two disciplines, and namely on the level of the annotation schemes provided by conversation analysis transcription methods. First, we demonstrate that the need for a stable structure in establishing an interaction scenario and in designing robot behaviours does not prevent the emergence of ordinariness or creativity among the participants engaged in this interaction. Secondly, based on an actual case, we emphasize the possibility of systematicness in CA transcription to support the choice (a) of the categories targeted by prediction methods and defined by the annotation scheme, and (b) of the verbal and non-verbal features used to create prediction models.
Article outline
  • 1.Interaction scenarios in social robotics and the notion of context in CA
    • a.Social robotics scenarios and the CA perspective
    • b.Emergence, spontaneity, stability
  • 2.Affective computing annotation of recordings in social robotics vs. CA transcription
    • a.Categories vs. emergence principle: Methodological comparison
    • b.Categories vs. emergence principles: Comparison of two types of production
    • c.Segmentation
  • 3.Conclusion and prospects
  • Acknowledgements
  • Notes
  • References
References (59)
References
Bartneck, C., Belpaeme, T., Eyssel, F., Kanda, T., Keijsers, M., & Šabanović, S. (2019). Human-robot interaction: An introduction. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Beach, W. A. (1993). Transitional regularities for ‘casual’ “Okay” usages. Journal of Pragmatics, 191, 325–352. Google Scholar logo with link to Google Scholar
Ben-Youssef, A., Clavel, C., Essid, S., Bilac, M., Chamoux, M., & Lim, A. (2017). UE- HRI: a new dataset for the study of user engagement in spontaneous human- robot interactions. Paper presented at the 19th ACM International Conference on Multimodal Interaction, Glasgow, UK.
Breazeal, C. (2003). Toward sociable robots. Robotics and Autonomous Systems, 421, 167–175. Google Scholar logo with link to Google Scholar
Button, G. (1991). Conversation-in-a-series. In D. Boden & D. H. Zimmerman (Eds.), Talk and social structure. Studies in Ethnomethodology and Conversation Analysis (pp. 251–277). Cambridge: Polity Press.Google Scholar logo with link to Google Scholar
Campano, S., Clavel, C., & Pélachaud, C. (2015). “I like this painting too”: when an ECA shares appreciations to engage users. Paper presented at the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15), Istanbul, Turkey.
Cassell, J., Torres, O., & Prevost, S. (1999). Turn taking vs. discourse structure: How best to model multimodal conversation. In Y. Wilks (Ed.), Machine Conversations (pp. 143–154). The Hague: Kluwer. Google Scholar logo with link to Google Scholar
Chapman, D. (1992). Computer rules, conversational rules. Computational Linguistics, 18(4), 531–536.Google Scholar logo with link to Google Scholar
Clavel, C., Vasilescu, I., & Devillers, L. (2011). Fiction support for realistic portrayals of fear-type emotional manifestations. Computer Speech & Language, 25(1), 63–83. Google Scholar logo with link to Google Scholar
Clavel, C., Cafaro, A., Campano, S., & Pelachaud, C. (2016). Fostering User Engagement in Face-to-Face Human-Agent Interactions: A Survey. In A. Esposito & L. C. Jain (Eds.), Toward Robotic Socially Believable Behaving Systems – Volume II: Modeling Social Signals (pp. 93–120). Cham: Springer International Publishing. Google Scholar logo with link to Google Scholar
Clavel, C., & Callejas, Z. (2016). Sentiment analysis: from opinion mining to human-agent interaction. IEEE Transactions on affective computing, 7(1), 74–93. Google Scholar logo with link to Google Scholar
Cowie, R., & Cornelius, R. R. (2003). Describing the emotional states that are expressed in speech. Speech communication, 40(1–2), 5–32. Google Scholar logo with link to Google Scholar
Dautenhahn, K. (2007). Socially intelligent robots: dimensions of human-robot interaction. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 362(1480), 679–704. Google Scholar logo with link to Google Scholar
Duranti, A. (1997). Linguistic anthropology. New York: Cambridge University Press. Google Scholar logo with link to Google Scholar
Duranti, A., & Goodwin, C. (1992). Rethinking context: an introduction. In A. Duranti & C. Goodwin (Eds.), Rethinking Context, language as an interactive phenomenon (pp. 1–42). Cambridge University Press.Google Scholar logo with link to Google Scholar
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and Autonomous Systems, 42(3–4), 143–166. Google Scholar logo with link to Google Scholar
Ford, C., & Thompson, S. (1996). Interactional units in conversation: syntactic, intonational, and pragmatic resources for the management of turns. In E. Ochs, E. Schegloff, & S. Thompson (Eds.), Interaction and Grammar (134-184): Cambridge University Press. Google Scholar logo with link to Google Scholar
Garfinkel, H. (1967). Studies in ethnomethodology. Engelwood Cliffs: Prentice-Hall.Google Scholar logo with link to Google Scholar
Goffman, E. (1973). La mise en scène de la vie quotidienne (Vol. 21. Les relations en public). Paris: Les Editions de minuit.Google Scholar logo with link to Google Scholar
(1983). The interaction order. American Sociological Review, 48(1), pp: 1–17. Google Scholar logo with link to Google Scholar
Goodwin, C. (1981). Conversational organization: interaction between speakers and hearers. New York: Academic Press.Google Scholar logo with link to Google Scholar
Heritage, J. (1991). L’Ethnométhodologie : une approche procédurale de l’action et de la communication. Réseaux CNET, 501, 89–130. Google Scholar logo with link to Google Scholar
Jefferson, G. (1978). Sequential aspects of storytelling in conversation. In J. Schenkein (Ed.), Studies in the organization of conversational interaction (pp. 219–248). New York: New York Academic Press. Google Scholar logo with link to Google Scholar
Jones, R. A. (2017). What makes a robot “social”? Social Studies of Science, 47(4), 556–579. Google Scholar logo with link to Google Scholar
Langlet, C., & Clavel, C. (2014). Modelling user’s attitudinal reactions to the agent utterances: focus on the verbal content. Paper presented at the 5th International Workshop on Corpora for Research on Emotion, Sentiment & Social Signals (ES3 2014), Reykjavik, Iceland.
(2018). Detecting User’s Likes and Dislikes for a Virtual Negotiating Agent. Paper presented at the 20th ACM International Conference on Multimodal Interaction, Boulder, USA.
Levinson, S. C. (1983). Pragmatics. Cambridge University Press. Google Scholar logo with link to Google Scholar
Licoppe, C., & Figeac, J. (2014). L’Organisation temporelle des engagements visuels dans des situations de multi-activité équipée en milieu urbain. Activités, 11(1).Google Scholar logo with link to Google Scholar
Licoppe, C., & Rollet, N. (2020, in press). « Je dois y aller ». Analyses de séquences de clôtures entre humains et robot. Réseaux. La Découverte. Google Scholar logo with link to Google Scholar
Mondada, L. (2006). Participants’ online analysis and multimodal practices: projecting the end of the turn and the closing of the sequence. Discourse Studies, 8(1), pp. 117–129. Google Scholar logo with link to Google Scholar
(2008). Documenter l’articulation des ressources multimodales dans le temps : la transcription d’enregistrements vidéos d’interactions. In M. Bilger (Ed.), Données orales. Les enjeux de la transcription (Vol. 371), pp. 127–156. Perpignan: Presses Universitaires de Perpignan.Google Scholar logo with link to Google Scholar
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2012). Foundations of machine learning. MIT press.Google Scholar logo with link to Google Scholar
Ochs, E. (1979). Transcription as a theory. In B. B. Schieffelin (Ed.), Developmental Pragmatics (pp. 42–72). Academic Press.Google Scholar logo with link to Google Scholar
Pelachaud, C., & Glas, N. (2015a). Definitions of Engagement in Human-Agent Interaction. Paper presented at the International Workshop on Engagement in Human Computer Interaction (ENHANCE), Xi’an, China.
(2015b). Topic transition strategies for an information-giving agent. Paper presented at the 15th European Workshop on natural Language Generation, Brighton, UK.
Pelikan, H. R. M., & Broth, M. (2016). Why That Nao?: How Humans Adapt to a Conventional Humanoid Robot in Taking Turns-at-Talk. Paper presented at the 2016 CHI Conference on Human Factors in Computing Systems, San Jose, California, USA.
Pitsch, K., Kuzuoka, H., Suzuki, Y., Süssenbach, L., Luff, P., & Heath, C. (2009). “The first five seconds”: Contingent stepwise entry into an interaction as a means to secure sustained engagement in Human-Robot-Interaction. Paper presented at the IEEE International Symposium on Robot and Human Interactive Communication ROMAN 2009, Toyama, Japan.
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018). Voice Interfaces in Everyday Life. Montreal QC, Canada: Association for Computing Machinery. Google Scholar logo with link to Google Scholar
Rollet, N. (2010). All the things you are. Activité multimodale, frontières et musiques improvisées en répétition In N. Andrieux-Reix (Ed.), Frontières. Du linguistique au sémiotique (pp. 279–302). Limoges: Lambert-Lucas.Google Scholar logo with link to Google Scholar
(2013). “D’accord”. Approche conversationnelle et multimodale d’une forme située dans les appels au Samu-Centre 15. L’Information grammaticale, 1391.Google Scholar logo with link to Google Scholar
Rollet, N., Jain, V., Licoppe, C., & Devillers, L. (2017). Towards Interactional Symbiosis: Epistemic Balance and Co-presence in a Quantified Self Experiment. In L. Gamberini, A. Spagnolli, G. Jacucci, B. Blankertz, & J. Freeman (Eds.), Symbiotic Interaction: 5th International Workshop, Symbiotic 2016, Padua, Italy, September 29–30, 2016, Revised Selected Papers (pp. 143–154). Cham: Springer International Publishing. Google Scholar logo with link to Google Scholar
Sacks, H. (1984). On doing ‘being ordinary’. In J. M. Atkinson & J. Heritage (Eds.), Structures of Social Action (pp. 413–429). Cambridge: CUP.Google Scholar logo with link to Google Scholar
(1992). Lectures on conversation (Jefferson, G. ed.). Oxford: Blackwell.Google Scholar logo with link to Google Scholar
Sacks, H., & Schegloff, E. (1973). Opening up closings. Semiotica, 81, 289–326.Google Scholar logo with link to Google Scholar
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A symplest systematics for the organization of turn-taking for conversation. Language, 501, 696–731. Google Scholar logo with link to Google Scholar
Sadazuka, K., Kuno, Y., Kawashima, M., & Yamazaki, K. (2007). Museum Guide Robot with Effective Head Gestures. Paper presented at the International Conference on Control, Automation and Systems, Seoul, Korea.
Schegloff, E. (1987). Between micro and macro: contexts and other connections. In B. Giesen, J. C. Alexander, R. Münch, & N. J. Smelser (Eds.), The Micro-macro link (pp. 207–234). Los Angeles: University of California Press.Google Scholar logo with link to Google Scholar
(2002). Accounts of conduct in interaction. Interruption, overlap, and turn-taking. In T. J. H. (Ed.), Handbook of sociological theory (pp. 287–321). New York: Kluwer Academic/Plenum publishers.Google Scholar logo with link to Google Scholar
(2007). Sequence organization in interaction. A Primer in Conversation Analysis. (Vol. 11): Cambridge University Press. Google Scholar logo with link to Google Scholar
Sidner, C. L., & Dzikovska, M. (2002, 16–16 Oct. 2002). Human-robot interaction: engagement between humans and robots for hosting activities. Paper presented at the 4th IEEE International Conference on Multimodal Interfaces, Pittsburgh, USA.
Sidner, C. L., Lee, C., Kidd, C. D., & Rich, C. (2005). Explorations in engagement for humans and robots. Artificial Intelligence, 1661, 140–164. Google Scholar logo with link to Google Scholar
Stivers, T. (2008). Stance, Alignment, and Affiliation During Storytelling: When Nodding Is a Token of Affiliation. Research on Language & Social Interaction, 44(1), 31–57. Google Scholar logo with link to Google Scholar
(2015). Coding Social Interaction: A Heretical Approach in Conversation Analysis? Research on Language and Social Interaction, 48(1), 1–19. Google Scholar logo with link to Google Scholar
Šabanović, S., & Chang, W.-L. (2016). Socializing robots: constructing robotic sociality in the design and use of the assistive robot PARO. AI and Society, 31(4), 537–551. Google Scholar logo with link to Google Scholar
Straub, I. (2016). “It looks like a human!” The interrelation of social presence, interaction and agency ascription: a case study about the effects of an android robot on social agency ascription. AI and Society, 31(4), 553–571. Google Scholar logo with link to Google Scholar
Suchman, L. (2007). Human-Machine Reconfigurations. Plans and situated actions, 2nd edition. New York: Cambridge University Press.Google Scholar logo with link to Google Scholar
Yu, Z., Scherer, S., Devault, D., Gratch, J., Stratou, G., Morency, L., & Cassell, J. (2013). Multimodal Prediction of Psychological Disorder: Learning Verbal and Nonverbal Commonality in Adjacency Pairs. Paper presented at the 17th Workshop Series on the Semantics and Pragmatics of Dialogue, Amsterdam, Netherland.
Zimmerman, D. (2006). How closing matters in emergency telephone calls. Paper presented at the Annual meeting of the American Sociology Association, Montréal, Canada.
Cited by (8)

Cited by eight other publications

Mlynář, Jakub, Lynn de Rijk, Andreas Liesenfeld, Wyke Stommel & Saul Albert
2025. AI in situated action: a scoping review of ethnomethodological and conversation analytic studies. AI & SOCIETY 40:3 ► pp. 1497 ff. DOI logo
Pollak, Anita, Elżbieta Biolik & Agata Chudzicka‐Czupała
2025. New Luddites? Counterproductive Work Behavior and Its Correlates, Including Work Characteristics, Stress at Work, and Job Satisfaction Among Employees Working With Industrial and Collaborative Robots. Human Factors and Ergonomics in Manufacturing & Service Industries 35:4 DOI logo
Rudaz, Damien
2025. The (Ir)relevance of Ethnomethodology and Conversation Analysis for Technology Companies: Incommensurability in Action. Human Studies DOI logo
Rudaz, Damien
2025. Social robots as designed artifacts: the impact of programming tools on “human–robot interaction”. AI & SOCIETY DOI logo
Dubois-Sage, Marion, Baptiste Jacquet, Frank Jamet & Jean Baratgin
2024. People with Autism Spectrum Disorder Could Interact More Easily with a Robot than with a Human: Reasons and Limits. Behavioral Sciences 14:2 ► pp. 131 ff. DOI logo
Lala, Divesh, Koji Inoue & Tatsuya Kawahara
2024. 2024 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), ► pp. 1 ff. DOI logo
Leo-Liu, Jindong
2023. Loving a “defiant” AI companion? The gender performance and ethics of social exchange robots in simulated intimate interactions. Computers in Human Behavior 141 ► pp. 107620 ff. DOI logo
Kopp, Stefan & Nicole Krämer
2021. Revisiting Human-Agent Communication: The Importance of Joint Co-construction and Understanding Mental States. Frontiers in Psychology 12 DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue