Cover not available

Article In: Group Dynamics in Human–Robot Interaction
Edited by Alessandra Sciutti, Dario Pasquali, Giulia Belgiovine and Linda Lastrico
[Interaction Studies 26:3] 2025
► pp. 392–421

Moderating multi-party conversations with social robots

Design and evaluation of control policies

ORCiD logo with link | University of Genoa
 | University of Genoa
 | University of Genoa
This content is being prepared for publication; it may be subject to changes.
Abstract
Social robotics is a multidisciplinary field focused on designing and implementing robots capable of interacting with humans in social environments. However, group conversations challenge robots in interpreting social signals for effective participation. This study evaluates control policies for moderating multi-party conversation dynamics using a humanoid robot. The system employs a cloud-based framework to calculate speaker dominance as a weighted combination of speaking time and word count, while the Louvain algorithm identifies subgroups among participants. Control policies aim to minimize dominance disparities and subgroup formation, fostering balanced participation and group cohesion. A study with 300 middle school students compared these policies to a baseline in which the robot did not address individuals directly. The results demonstrated that the proposed policies reduced dominance gaps and subgroup formation, promoting more balanced interactions. These findings highlight the potential applicability of the approach across education, healthcare, and entertainment.
Article outline
  • 1.Introduction
  • 2.Related work
    • 2.1Participant recognition
    • 2.2Engagement evaluation
    • 2.3Engagement and turn-taking management
    • 2.4Dominance estimation
    • 2.5Subgroup recognition
    • 2.6Moderating and facilitating
  • 3.System architecture and control policies
    • 3.1System architecture
    • 3.2Control policies
      • 3.2.1Balancing policy
      • 3.2.2Community policy
      • 3.2.3Hard and soft versions of policies
  • 4.Materials and methods
    • 4.1Participants
    • 4.2Hypotheses
    • 4.3Conditions
    • 4.4Experimental procedure
    • 4.5Measurements
  • 5.Results
    • 5.1Dominance
    • 5.2Communities
    • 5.3Testing hypothesis H1
    • 5.4Testing hypothesis H2
    • 5.5Discussion
    • 5.6Limitations
    • 5.7Future works
  • 6.Conclusions
  • Note
  • References
References (66)
References
Addlesee, A., Cherakara, N., Nelson, N., Hernández Garcia, D., Gunson, N., Sieińska, W., Romeo, M., Dondrup, C., & Lemon, O. (2024). A multiparty conversational social robot using llms. Companion of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRl), 1273–1275. Google Scholar logo with link to Google Scholar
Addlesee, A., Sieińska, W., Gunson, N., Hernández Garcia, D., Dondrup, C., & Lemon, O. (2023). Multi-party goal tracking with llms: Comparing pre-training, fine-tuning, and prompt engineering. Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), 229–241. Google Scholar logo with link to Google Scholar
Aliasghari, P., Taheri, A., Meghdari, A., & E., M. (2020). Implementing a gaze control system on a social robot in multi-person interactions. SN Applied Sciences, 21, 1–13. Google Scholar logo with link to Google Scholar
Axelsson, M., Spitale, M., & Gunes, H. (2023). Robotic coaches delivering group mindfulness practice at a public cafe. HRI’23, 86–90. Google Scholar logo with link to Google Scholar
Bales, R. F. (1950). Interaction process analysis: A method for the study of small groups. Addison-Wesley.Google Scholar logo with link to Google Scholar
Ban, Y., Alameda-Pineda, X., Badeig, F., Ba, S., & Horaud, R. (2017). Tracking a varying number of people with a visually-controlled robotic head. IROS 2017, 4144–4151. Google Scholar logo with link to Google Scholar
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., & Tanaka, F. (2018). Social robots for education: A review. Science Robotics, 3 (21), eaat5954. Google Scholar logo with link to Google Scholar
Bi, J., Hu, F., Wang, Y., Luo, M., & He, M. (2023a). Human engagement intention intensity recognition method based on two states fusion fuzzy inference system. Intelligent Service Robotics, 121, 307–322. Google Scholar logo with link to Google Scholar
(2023b). A method based on interpretable machine learning for recognizing the intensity of human engagement intention. Scientific Reports, 131, 1–14. Google Scholar logo with link to Google Scholar
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems.Google Scholar logo with link to Google Scholar
Burgoon, J. K., & Dunbar, N. E. (2006). Nonverbal expressions of dominance and power in human relationships. In The sage handbook of nonverbal communication (pp. 279–297). Sage Publications, Inc. Google Scholar logo with link to Google Scholar
Chew, J. Y., & Nakamura, K. (2023). Who to teach a robot to facilitate multiparty social interactions? Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (HRI), 51. Google Scholar logo with link to Google Scholar
Correia, F., Campos, J., Melo, F., & Paiva, A. (2023). Robotic gaze responsiveness in multiparty teamwork. International Journal of Social Robotics, 151, 27–36. Google Scholar logo with link to Google Scholar
Cumbal, R., Kazzi, D. A., Winberg, V., & Engwall, O. (2022). Shaping unbalanced multi-party interactions through adaptive robot backchannels. Proc. IVA’22, 1–7. Google Scholar logo with link to Google Scholar
Davison, D. P., Wijnen, F. M., Charisi, V., van der Meij, J., Evers, V., & Reidsma, D. (2020). Working with a social robot in school: A longterm real-world unsupervised deployment. Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 63–72. Google Scholar logo with link to Google Scholar
De Ruiter, J.-P., Mitterer, H., & Enfield, N. J. (2006). Projecting the end of a speaker’s turn: A cognitive cornerstone of conversation. Language, 821, 515–535. Google Scholar logo with link to Google Scholar
Dunbar, N. E., & Burgoon, J. K. (2005). Perceptions of power and interactional dominance in interpersonal relationships. Journal of Social and Personal Relationships, 22 (2), 207–233. Google Scholar logo with link to Google Scholar
Ellyson, S. L., & Dovidio, J. F. (1985). Power dominance and nonverbal behavior. Springer. Google Scholar logo with link to Google Scholar
Escobar-Planas, M., Charisi, V., & Gomez, E. (2022). “That robot played with us!” children’s perceptions of a robot after a child-robot group interaction. Proceedings of the ACM on Human-Computer Interaction, 61, 1–23. Google Scholar logo with link to Google Scholar
Forsyth, D. R. (2018). Group dynamics (7th). Cengage Learning.Google Scholar logo with link to Google Scholar
Foster, M. E., Gaschler, A., Giuliani, M., Isard, A., Pateraki, M., & Petrick, R. P. (2012). Two people walk into a bar: Dynamic multi-party social interaction with a robot agent. ICMI’12, 3–10. Google Scholar logo with link to Google Scholar
Foster, M., Gaschler, A., & Giuliani, M. (2017). Automatically classifying user engagement for dynamic multi-party human-robot interaction. International Journal of Social Robotics, 91, 659–674. Google Scholar logo with link to Google Scholar
Fraune, M. R., Šabanović, S., & Kanda, T. (2019). Human group presence, group characteristics, and group norms affect human-robot interaction in naturalistic settings. Frontiers in Robotics and AI, 61 (JUN). Google Scholar logo with link to Google Scholar
Gillet, S., Cumbal, R., Pereira, A., Lopes, J., Engwall, O., & Leite, I. (2021). Robot gaze can mediate participation imbalance in groups with different skill levels. HRI’21, 303–311. Google Scholar logo with link to Google Scholar
Gillet, S., Parreira, M. T., Vázquez, M., & Leite, I. (2022). Learning gaze behaviors for balancing participation in group human-robot interactions. HRI’22, 265–274. Google Scholar logo with link to Google Scholar
Gillet, S., Vázquez, M., Peters, C., Yang, F., & Leite, I. (2022). Multiparty interaction between humans and socially interactive agents. In The handbook on socially interactive agents: 20 years of research on embodied conversational agents, intelligent virtual agents, and social robotics volume 2: Interactivity, platforms, application (pp. 113–154). ACM. Google Scholar logo with link to Google Scholar
Gonzalez, J., Belgiovine, G., Sciutti, A., Sandini, G., & Rea, F. (2021). Towards a cognitive framework for multimodal person recognition in multiparty hri. HAI’21, 412–416. Google Scholar logo with link to Google Scholar
Grassi, L., Recchiuto, C. T., & Sgorbissa, A. (2021). Cloud services for social robots and artificial agents. Proc. AIRO 2021, 1–6.Google Scholar logo with link to Google Scholar
(2022). Knowledge-grounded dialogue flow management for social robots and conversational agents. International Journal of Social Robotics, 1–21. Google Scholar logo with link to Google Scholar
(2024). Enhancing llm-based human-robot interaction with nuances for diversity awareness. [URL].
Hung, H., & Gatica-Perez, D. (2010). Estimating cohesion in small groups using audio-visual nonverbal behavior. IEEE Transactions on Multimedia, 12 (6), 563–575. Google Scholar logo with link to Google Scholar
Kim, J., Yun, S.-S., Kang, B.-N., Kim, D., & Choi, J. (2017). Reliable multiperson identification using dcnn-based face recognition algorithm and scale-ratio method. Proc. URAI 2017, 97–101. Google Scholar logo with link to Google Scholar
Klotz, D., Wienke, J., Peltason, J., Wrede, B., Wrede, S., Khalidov, V., & Odobez, J.-M. (2011). Engagement-based multi-party dialog with a humanoid robot. SIGDIAL’11, 341–343.Google Scholar logo with link to Google Scholar
Li, L., Yu, X., Li, J., Wang, G., Shi, J.-Y., Tan, Y. K., & Li, H. (2012). Visionbased attention estimation and selection for social robot to perform natural interaction in the open world. HRI’12, 183–184.Google Scholar logo with link to Google Scholar
Lin, W., Li, Y., Xiao, H., See, J., Zou, J., Xiong, H., Wang, J., & Mei, T. (2021). Group reidentification with multigrained matching and integration. IEEE Transactions on Cybernetics, 51 (3), 1478–1492. Google Scholar logo with link to Google Scholar
Ma, F., Ma, Z., Sun, B., & Li, S. (2022). TA-CNN: A unified network for human behavior analysis in multi-person conversations. MM ’22, 7099–7103. Google Scholar logo with link to Google Scholar
Massé, B., Ba, S., & Horaud, R. (2018). Tracking gaze and visual focus of attention of people involved in social interaction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 (11), 2711–2724. Google Scholar logo with link to Google Scholar
Matsuyama, Y., Akiba, I., S., F., & T., K. (2015). Four-participant group conversation. Computer Speech & Language, 33 (1), 1–24. Google Scholar logo with link to Google Scholar
Mongile, S., Pusceddu, G., Cocchella, F., Lastrico, L., Belgiovine, G., Tanevska, A., Rea, F., & Sciutti, A. (2023). What if a social robot excluded you? using a conversational game to study social exclusion in teen-robot mixed groups. HRI’23, 208–212. Google Scholar logo with link to Google Scholar
Moujahid, M., Wilson, B., Hastie, H., & Lemon, O. (2022). Demonstration of a robot receptionist with multi-party situated interaction. HRI’22, 1202–1203. Google Scholar logo with link to Google Scholar
Murali, P., Steenstra, I., Yun, H. S., Shamekhi, A., & Bickmore, T. (2023). Improving multiparty interactions with a robot using large language models. CHI EA ’23, 1–8. Google Scholar logo with link to Google Scholar
Mussakhojayeva, S., Zhanbyrtayev, M., Agzhanov, Y., & Sandygulova, A. (2016). Who should robots adapt to within a multi-party interaction in a public space? 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI), 483–484. Google Scholar logo with link to Google Scholar
Neto, I., Correia, F., Rocha, F., Piedade, P., Paiva, A., & Nicolau, H. (2023). The robot made us hear each other: Fostering inclusive conversations among mixed-visual ability children. HRI’23, 13–23. Google Scholar logo with link to Google Scholar
Pappu, A., Sun, M., Sridharan, S., & Rudnicky, A. (2013). Situated multiparty interaction between humans and agents. In Hci’13 (pp. 107–116). Springer. Google Scholar logo with link to Google Scholar
Pereira, A., Prada, R., & Paiva, A. (2014). Improving social presence in human-agent interaction. Proc, CHI ’14, 1449–1458. Google Scholar logo with link to Google Scholar
Que, X., Checconi, F., Petrini, F., & Gunnels, J. A. (2015). Scalable community detection with the louvain algorithm. IPDPS 2015, 28–37. Google Scholar logo with link to Google Scholar
Recchiuto, C., Gava, L., Grassi, L., Grillo, A., Lagomarsino, M., Lanza, D., Liu, Z., Papadopoulos, C., Papadopoulos, I., Scalmato, A., et al. (2020). Cloud services for culture aware conversation: Socially assistive robots and virtual assistants. Proc. UR’20, 270–277.Google Scholar logo with link to Google Scholar
Recchiuto, C. T., & Sgorbissa, A. (2020). A feasibility study of culture-aware cloud services for conversational robots. IEEE Robotics and Automation Letters, 5 (4), 6559–6566. Google Scholar logo with link to Google Scholar
Schmid Mast, M. (2002). Dominance as expressed and inferred through speaking time: A meta-analysis. Human Communication Research, 28 (3), 420–450.Google Scholar logo with link to Google Scholar
Sebo, S., Dong, L. L., Chang, N., Lewkowicz, M., Schutzman, M., & Scassellati, B. (2020). The influence of robot verbal support on human team members: Encouraging outgroup contributions and suppressing ingroup supportive behavior. Frontiers in Psychology, 111, 1–16. Google Scholar logo with link to Google Scholar
Shintani, T., Ishi, C. T., & Ishiguro, H. (2021). Analysis of role-based gaze behaviors and gaze aversions, and implementation of robot’s gaze control for multi-party dialogue. HAI’21, 332–336. Google Scholar logo with link to Google Scholar
Short, E., & Mataric, M. J. (2017). Robot moderation of a collaborative game: Towards socially assistive robotics in group interactions. ROMAN 2017, 385–390. Google Scholar logo with link to Google Scholar
Short, E. S., Sittig-Boyd, K., & Matarić, M. J. (2016). Modeling moderation for multi-party socially assistive robotics. RO-MAN 2016.Google Scholar logo with link to Google Scholar
Skantze, G. (2021). Turn-taking in conversational systems and human-robot interaction: A review. Computer Speech & Language, 671, 1–26. Google Scholar logo with link to Google Scholar
Skantze, G., Johansson, M., & Beskow, J. (2015). Exploring turn-taking cues in multi-party human-robot discussions about objects. ICMI’15, 67–74. Google Scholar logo with link to Google Scholar
Tatarian, K., Chamoux, M., Pandey, A. K., & Chetouani, M. (2021). Robot gaze behavior and proxemics to coordinate conversational roles in group interactions. RO-MAN 2021, 1297–1304. Google Scholar logo with link to Google Scholar
Taylor, A., & Riek, L. D. (2022). Regroup: A robot-centric group detection and tracking system. HRI’22, 412–421. Google Scholar logo with link to Google Scholar
Telisheva, Z., Zhanatkyzy, A., Oralbayeva, N., Amirova, A., Aimysheva, A., & Sandygulova, A. (2022). The effects of dyadic vs triadic interaction on children’s cognitive and affective gains in robot-assisted alphabet learning. ICSR 2022, 204–213. Google Scholar logo with link to Google Scholar
Trombly, M., Shahverdi, P., Huang, N., Chen, Q., Korneder, J., & Louie, W.-Y. G. (2022). Robot-mediated group instruction for children with asd: A pilot study. RO-MAN 2022, 1506–1513. Google Scholar logo with link to Google Scholar
Tuncer, S., Gillet, S., & Leite, I. (2022). Robot-mediated inclusive processes in groups of children: From gaze aversion to mutual smiling gaze. Frontiers in Robotics and AI, 91, 1–15. Google Scholar logo with link to Google Scholar
Xu, Q., Li, L., & Wang, G. (2013). Designing engagement-aware agents for multiparty conversations. CHI’13, 2233–2242. Google Scholar logo with link to Google Scholar
Yatsushiro, M., Ikeda, N., Hayashi, Y., & Nakano, Y. I. (2013). A dominance estimation mechanism using eye-gaze and turn-taking information. Proc. GazeIn ’13, 13–18. Google Scholar logo with link to Google Scholar
Yoshino, T., Takase, Y., & Nakano, Y. I. (2015). Controlling robot’s gaze according to participation roles and dominance in multiparty conversations. HRI’15, 127–128. Google Scholar logo with link to Google Scholar
Yumak, Z., & Magnenat-Thalmann, N. (2015). Multimodal and multi-party social interactions. Springer. Google Scholar logo with link to Google Scholar
Żarkowski, M. (2019). Multi-party turn-taking in repeated human-robot interactions: An interdisciplinary evaluation. International Journal of Social Robotics, 111, 693–707. Google Scholar logo with link to Google Scholar
Zhang, Z., Zheng, J., & Thalmann, N. M. (2021). Engagement intention estimation in multiparty human-robot interaction. RO-MAN 2021, 117–122. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue