Cover not available

Article published In: Human Robot Collaborative Intelligence: Theory and applications
Edited by Chenguang Yang, Xiaofeng Liu, Junpei Zhong and Angelo Cangelosi
[Interaction Studies 20:1] 2019
► pp. 4–24

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Temporal patterns in multi-modal social interaction between elderly users and service robot

 | University of the West of England
 | Sheffield Hallam University
 | University of Manchester
 | Plymouth University
Published online: 15 July 2019
DOI logo with link
https://doi.org/10.1075/is.18042.wan
Abstract
Social interaction, especially for older people living alone is a challenge currently facing human-robot interaction (HRI). There has been little research on user preference towards HRI interfaces. In this paper, we took both objective observations and participants’ opinions into account in studying older users with a robot partner. The developed dual-modal robot interface offered older users options of speech or touch screen to perform tasks. Fifteen people aged from 70 to 89 years old, participated. We analyzed the spontaneous actions of the participants, including their attentional activities and conversational activities, the temporal characteristics of these social behaviours, as well as questionnaires. It has been revealed that social engagement with the robot demonstrated by older people was no different from what might be expected towards a human partner. This study is an early attempt to reveal the social connections between human beings and a personal robot in real life.
Article outline
  • I.Introduction
    • A.Elderly people-robots interaction
    • B.User interfaces
    • C.Research highlights
  • II.User platform for Robot-Era
  • III.Experiments and data
    • A.Participants
    • B.Experimental setup
    • C.Procedure and task
    • D.Data characteristics and annotation
  • IV.Results
    • A.Attentional behaviours – Where to look at?
    • B.Conversational behaviours – Speak or not?
    • C.Preference towards dual-modalities: “Talking” or “reading”?
    • D.PoIs
    • E.Questionnaire opinions
  • V.Conclusions
  • Acknowledgments
  • Notes
  • References
  • Sources
References (27)
References
Al-Razgan, M., H. Al-Khalifa, M. Al-Shahrani, and H. Al-Ajmi. (2012). Touch-based mobile phone interface guidelines and design recommendations for elderly people: A survey of the literature, ser. Lecture Notes in Computer Science. Berlin Heidelberg: Springer, vol. 76661, pp. 568–574.Google Scholar logo with link to Google Scholar
Blythe, M. A., A. F. Monk, and K. Doughty. (2005). “Socially dependable design: The challenge of ageing populations for HCI,” Interacting with Computers, vol. 17, no. 6, pp. 672–689. Google Scholar logo with link to Google Scholar
Bohus, D. and A. Rudnicky. (2009). “The ravenclaw dialog management framework: Architecture and systems,” Computer Speech and Language, vol. 23, no. 3, pp. 332–361. Google Scholar logo with link to Google Scholar
Breazealm C., C. Kidd, A. Thomaz, G. Hoffman, and M. Berlin. (2005). “Effects of nonverbal communication on efficiency and robustness in human-robot teamwork,” in Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 708–713.Google Scholar logo with link to Google Scholar
Cavallo, F., R. Esposito, R. Limosani, A. Manzi, R. Bevilacqua, E. Felici, A. D. Nuovo, A. Cangelosi, F. Lattanzio, and P. Dario. (2018). “Robotic services acceptance in smart environments with older adults: User satisfaction and acceptability study,” J Med Internet Res, vol. 20, no. 9. Google Scholar logo with link to Google Scholar
Di Nuovo, A., F. Broz, T. Belpaeme, A. Cangelosi, F. Cavallo, R. Esposito, and P. Dario. (2014). “A web based multi-modal interface for elderly users of the Robot-Era multi-robot services,” in Proc. IEEE International Conference on Systems, Man and Cybernetics, pp. 2186–2191.Google Scholar logo with link to Google Scholar
Hare, B. and M. Tomasello. (2005). “Human-like social skills in dogs?” Trends in Cognitive Sciences, vol. 9, no. 9, pp. 439–444. Google Scholar logo with link to Google Scholar
Kipp, M.. (2014). Handbook of Corpus Phonology. Oxford University Press, ch. ANVIL: A Universal Video Research Tool, pp. 420–436.Google Scholar logo with link to Google Scholar
Knapp, M. L., J. A. Hall, and T. G. Horgan. (2013). Nonverbal communication in human interaction, 8th ed. Boston, USA: Wadsworth, 2013.Google Scholar logo with link to Google Scholar
Lemon, O.. (2004). “Context-sensitive speech recognition in Information-State Update dialogue systems: results for the grammar switching approach,” in Proc. Eighth Workshop on the Semantics and Pragmatics of Dialogue, pp. 49–55.Google Scholar logo with link to Google Scholar
Linert, J. and P. Kopacek. (2016). “Robots for education (edutainment),” in IFAC-PapersOnLine, Elsevier, Ed., vol. 49, no. 29, pp. 24–29.Google Scholar logo with link to Google Scholar
Mayer, P., C. Beck, and P. Panek. (2012). “Examples of multimodal user interfaces for socially assistive robots in ambient assisted environments,” in Proc. IEEE International Conference on Cognitive Infocommunications, pp. 401–406.Google Scholar logo with link to Google Scholar
Morency, L.-P. and T. Darrell. (2004). “From conversational tooltips to grounded discourse: Head post tracking in interactive dialog systems,” in ACM International Conference on Multimodal Interaction, pp. 32–37.Google Scholar logo with link to Google Scholar
Mutlu, B., T. Shiwa, T. Kanda, H. Ishiguro, and N. Hagita. (2009). “Footing in human-robot conversations: How robots might shape participant roles using gaze cues,” in Proc. ACM/IEEE International Conference on Human Robot Interaction, pp. 61–68.Google Scholar logo with link to Google Scholar
Portet, F., M. Vacher, C. Golanski, C. Roux, and B. Meillon. (2013). “Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects,” Personal and Ubiquitous Computing, vol. 17, no. 1, pp. 127–144. Google Scholar logo with link to Google Scholar
“Robot-Era project: Implementation and integration of advanced robotic systems and intelligent environments in real scenarios for the ageing population,” FP7-ICT-Challenge 5: ICT for Health, Ageing Well, Inclusion and Governance. Grant agreement number 288899.
Schneider, J., S. Irgenfried, W. Stork, and H. Wörn. (2015). “A multimodal human machine interface for a robotic mobility aid,” in Proc. IEEE International Conference on Automation, Robotics and Applications, pp. 289–294.Google Scholar logo with link to Google Scholar
Sehili, M. A., F. Yang, and L. Devillers. (2014). “Attention detection in elderly people-robot spoken interaction,” in Proc. ICMI Workshop on Multimodal, Multi-Party, Real-World Human-Robot Interaction, pp. 7–12.Google Scholar logo with link to Google Scholar
Sheikhi, S. and J.-M. Odobez. (2014). “Combining dynamic head pose-gaze mapping with the robot conversational state for attention recognition in human-robot interactions,” Pattern Recognition Letters.Google Scholar logo with link to Google Scholar
Smarr, C., A. Prakash, J. Beer, T. Mitzner, C. Kemp, and W. Rogers. (2012). “Older adults preferences for and acceptance of robot assistance for everyday living tasks,” in Proc. Human Factors and Ergonomics Society Annual Meeting, pp. 153–157. Google Scholar logo with link to Google Scholar
Staudte, M. and M. Crocker. (2009). “Visual attention in spoken human-robot interaction,” in Proc. ACM/IEEE International Conference on Human-Robot Interaction, pp. 77–84.Google Scholar logo with link to Google Scholar
Tang, D., B. Yusuf, J. Botzheim, N. Kubota, and C. Chan. (2015). “A novel multimodal communication framework using robot partner for aging population,” Expert Systems with Applications, vol. 42, no. 9, pp. 4540–4555. Google Scholar logo with link to Google Scholar
United Nations. (2017, October). “Population facts.”Google Scholar logo with link to Google Scholar
Wang, N., F. Broz, A. Di Nuovo, T. Belpaeme, and A. Cangelosi. (2016). Recent Advances in Nonlinear Speech Processing, ser. 2190–3018. Springer International Publishing, vol. 481, ch. A user-centric design of service robots speech interface for the elderly, pp. 275–283. Google Scholar logo with link to Google Scholar
Yamazaki, A., K. Yamazaki, Y. Kuno, M. Burdelski, M. Kawashima, and H. Kuzuoka. (2008). “Precision timing in human-robot interaction: Coordination of head movement and utterance,” in Proc. SIGCHI Conference on Human Factors in Computing Systems, pp. 131–140.Google Scholar logo with link to Google Scholar
Yu, C., M. Scheutz, and P. Schermerhorn. (2010). “Investigating multimodal real-time patterns of joint attention in an HRI word learning task,” in Proc. ACM/IEEE International Conference on Human-Robot Interaction, pp. 309–316.Google Scholar logo with link to Google Scholar
Sources
(2015). [Online]. Available: [URL]
Cited by (14)

Cited by 14 other publications

Li, Yi, Yaping Chang, Tangwutu Yang & You Li
2025. Understanding consumer responses to robot roles in human-robot service collaboration. Journal of Retailing and Consumer Services 87 ► pp. 104407 ff. DOI logo
Maroto-Gómez, Marcos, María Malfaz, José Carlos Castillo, Álvaro Castro-González & Miguel Ángel Salichs
2025. Personalizing Activity Selection in Assistive Social Robots from Explicit and Implicit User Feedback. International Journal of Social Robotics 17:10 ► pp. 1999 ff. DOI logo
Wu, Yu-Dan, Jia-Xin Dong, Fu-Min Yu, Zhe-Hao Dong, Wei Ma, Yue Cai, Yu-Qing Cai, Yang Mu, Xiang Cui, Yi-Ran Wang, Hui-Jun Li, Xiao-Tao Yang, Duo-Ning Yuan, Shuang Wang, Nuo Cheng & Guang-Wei Zhang
2025. Mapping the global research landscape and trends of older people living alone: a bibliometric analysis. Frontiers in Aging 6 DOI logo
You, Soobin & Heejeong Choi
2025. A sociocultural perspective on AI assistive technology for older adults' social connectedness: A scoping review. Family Relations 74:3 ► pp. 1080 ff. DOI logo
Choi, Narae & Do-Hyung Park
2024. Proceedings of the 2024 7th International Conference on Machine Learning and Machine Intelligence (MLMI), ► pp. 163 ff. DOI logo
Teng, Lefa, Chuluo Sun, Yifei Chen, Michael W. Lever & Lianne Foti
2024. Partner or servant? The influence of robot role positioning on consumers’ brand evaluations. Journal of Business Research 180 ► pp. 114698 ff. DOI logo
Hu, Chih-Chien, Yu-Fen Yang & Nian-Shing Chen
2023. Human–robot interface design – the ‘Robot with a Tablet’ or ‘Robot only’, which one is better?. Behaviour & Information Technology 42:10 ► pp. 1590 ff. DOI logo
Zhang, Ying, Chenguang Yang, Sheng Xu & Yongsheng Ou
2023. Obstacle avoidance in human-robot cooperative transportation with force constraint. Science China Information Sciences 66:1 DOI logo
Blaurock, Marah, Martina Čaić, Mehmet Okan & Alexander P. Henkel
2022. Robotic role theory: an integrative review of human–robot service interaction to advance role theory in the age of social robots. Journal of Service Management 33:6 ► pp. 27 ff. DOI logo
Carradore, Marco
2022. People’s Attitudes Towards the Use of Robots in the Social Services: A Multilevel Analysis Using Eurobarometer Data. International Journal of Social Robotics 14:3 ► pp. 845 ff. DOI logo
Zhuang, Liang, Ching-Hsien Hsu & Priyan Malarvizhi Kumar
2022. RETRACTED ARTICLE: IoT Based Multimodal Social Interaction Activity Framework for the Physical Education System. Wireless Personal Communications 127:S1 ► pp. 41 ff. DOI logo
Conti, Daniela, Santo Di Nuovo & Alessandro Di Nuovo
2021. A Brief Review of Robotics Technologies to Support Social Interventions for Older Users. In Human Centred Intelligent Systems [Smart Innovation, Systems and Technologies, 189], ► pp. 221 ff. DOI logo
Wang, J., Y. Fu, V. Lou, S.Y. Tan & E. Chui
2021. A systematic review of factors influencing attitudes towards and intention to use the long-distance caregiving technologies for older adults. International Journal of Medical Informatics 153 ► pp. 104536 ff. DOI logo
Wang, Ning, Chuize Chen & Alessandro Di Nuovo
2021. A Framework of Hybrid Force/Motion Skills Learning for Robots. IEEE Transactions on Cognitive and Developmental Systems 13:1 ► pp. 162 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue