Cover not available

Article published In: Socially Acceptable Robot Behavior: Approaches for Learning, Adaptation and Evaluation
Edited by Oliver Roesler, Elahe Bagheri, Amir Aly, Silvia Rossi and Rachid Alami
[Interaction Studies 23:3] 2022
► pp. 427–468

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Learning social navigation from demonstrations with conditional neural processes

 | Bogazici University
 | Bogazici University
Published online: 21 April 2023
DOI logo with link
https://doi.org/10.1075/is.22018.yil
Abstract
Sociability is essential for modern robots to increase their acceptability in human environments. Traditional techniques use manually engineered utility functions inspired by observing pedestrian behaviors to achieve social navigation. However, social aspects of navigation are diverse, changing across different types of environments, societies, and population densities, making it unrealistic to use hand-crafted techniques in each domain. This paper presents a data-driven navigation architecture that uses state-of-the-art neural architectures, namely Conditional Neural Processes, to learn global and local controllers of the mobile robot from observations. Additionally, we leverage a state-of-the-art, deep prediction mechanism to detect situations not similar to the trained ones, where reactive controllers step in to ensure safe navigation. Our results demonstrate that the proposed framework can successfully carry out navigation tasks regarding social norms in the data. Further, we showed that our system produces fewer personal-zone violations, causing less discomfort.
Article outline
  • Introduction
  • Related work
    • Hybrid path planning
      • Global path planning
      • Local path planning
    • Social navigation
  • Our method
    • I – data-driven global controller
      • Training the data-driven global controller
      • Querying the trained data-driven global controller
    • II – data-driven local controller
      • Training the data-driven local controller
      • Querying the trained data-driven local controller
    • III – failure prediction module
    • IV – hand-crafted reactive controller
  • Experiments and results
    • Analysis of the generated global trajectories
    • Analysis of the local controller: Evasive maneuvers
    • Comparison of local controllers
    • Performance of the complete system
    • Contribution of the Failure Prediction Module
    • Scalability of CNP
  • Conclusion
  • Acknowledgements
  • References
References (75)
References
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S. (2016). Social lstm: Human trajectory prediction in crowded spaces. In 2016 ieee conference on computer vision and pattern recognition (cvpr) (p. 961–971). Google Scholar logo with link to Google Scholar
Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein gan.Google Scholar logo with link to Google Scholar
Asghari Oskoei, M., Walters, M., & Dautenhahn, K. (2010). An autonomous proxemic system for a mobile companion robot. In Proceedings of the aisb 2010 symposium on new frontiers for human robot interaction. Leicester, UK.Google Scholar logo with link to Google Scholar
Biswas, A., Wang, A., Silvera, G., Steinfeld, A., & Admoni, H. (2022). Socnavbench: A grounded simulation testing framework for evaluating social navigation. ACM Transactions on Human-Robot Interaction (THRI), 11(3), 1–24. Google Scholar logo with link to Google Scholar
Borenstein, J., Koren, Y., et al. (1991). The vector field histogram-fast obstacle avoidance for mobile robots. IEEE transactions on robotics and automation, 7(3), 278–288. Google Scholar logo with link to Google Scholar
Brooks, R. (1986). A robust layered control system for a mobile robot. IEEE journal on robotics and automation, 2(1), 1423. Google Scholar logo with link to Google Scholar
Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. ar Xiv preprint arXiv:1810.12894.Google Scholar logo with link to Google Scholar
Burgard, W., Cremers, A., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., … Thrun, S. (1999). Experiences with an interactive museum tour-guide robot. Artif. Intell., 1141, 3–55. Google Scholar logo with link to Google Scholar
Cai, K., Wang, C., Cheng, J., De Silva, C. W., & Meng, M. Q.-H. (2020). Mobile Robot Path Planning in Dynamic Environments: A Survey. arXiv preprint arXiv:2006.14195.Google Scholar logo with link to Google Scholar
Che, Y., Okamura, A. M., & Sadigh, D. (2020). Efficient and trustworthy social navigation via explicit and implicit robot-human communication. IEEE Transactions on Robotics, 36(3), 692–707. Google Scholar logo with link to Google Scholar
Chen, Y. F., Everett, M., Liu, M., & How, J. P. (2017). Socially aware motion planning with deep reinforcement learning. CoRR, abs/1703.08862. Retrieved from [URL].
Dudek, G., & Jenkin, M. (2010). Computational principles of mo-bile robotics. Cambridge university press. Google Scholar logo with link to Google Scholar
Farina, F., Fontanelli, D., Garulli, A., Giannitrapani, A., & Prat-tichizzo, D. (2017). Walking ahead: The headed social force model. PloS one, 12(1), e0169734. Google Scholar logo with link to Google Scholar
Ferrer, G., Garrell, A., & Sanfeliu, A. (2013). Robot companion: A social-force based approach with human awareness-navigation in crowded environments. In 2013 ieee/rsj international conference on intelligent robots and systems (pp. 1688–1694). Google Scholar logo with link to Google Scholar
Festo Robotics, R. (2020). Robotino 4: For research and education. Retrieved from [URL]
Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and autonomous systems, 42(3–4), 143–166. Google Scholar logo with link to Google Scholar
Fox, D., Burgard, W., & Thrun, S. (1997). The dynamic window approach to collision avoidance. IEEE Robotics Automation Magazine, 4(1), 23–33. Google Scholar logo with link to Google Scholar
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Sax-ton, D., Shanahan, M., … Eslami, S. M. A. (2018, 10–15 Jul). Conditional Neural Processes. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning (Vol. 801, pp. 1704–1713). PMLR. Retrieved from [URL]
Giesbrecht, J. (2004). Global path planning for unmanned ground vehicles (Tech. Rep.). Defence Research and Development Suffield (Alberta).Google Scholar logo with link to Google Scholar
Glasius, R., Komoda, A., & Gielen, S. C. (1995). Neural network dynamics for path planning and obstacle avoidance. Neural Networks, 8 (1), 125–133. Google Scholar logo with link to Google Scholar
Good fellow, I. (2016). Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.Google Scholar logo with link to Google Scholar
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y. (2014). Generative adversarial networks. arXiv preprint arXiv:1406.2661.Google Scholar logo with link to Google Scholar
Gordon, J., Bruinsma, W. P., Foong, A. Y., Requeima, J., Dubois, Y., & Turner, R. E. (2019). Convolutional conditional neural processes. arXiv preprint arXiv:1910.13556.Google Scholar logo with link to Google Scholar
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A. (2018). Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 2255–2264). Google Scholar logo with link to Google Scholar
Hall, E. (1966). The hidden dimension. New York, NY, US: Anchor Books.Google Scholar logo with link to Google Scholar
Helbing, D., & Molnar, P. (1995). Social force model for pedestrian dynamics. Physical review E, 51(5), 4282. Google Scholar logo with link to Google Scholar
Holtz, J., & Biswas, J. (2021). Socialgym: A framework for benchmarking social robot navigation. arXiv preprint arXiv:2109.11011.Google Scholar logo with link to Google Scholar
Huang, K.-C., Li, J.-Y., & Fu, L.-C. (2010). Human-oriented navigation for service providing in home environment. In Sice annual conference 2010, proceedings of (pp. 1892–1897). Taipei, Taiwan.Google Scholar logo with link to Google Scholar
Kambhampati, S., & Davis, L. (1986). Multiresolution path planning for mobile robots. IEEE Journal on Robotics and Automation, 2(3), 135–145. Google Scholar logo with link to Google Scholar
Karnan, H., Nair, A., Xiao, X., Warnell, G., Pirk, S., Toshev, A., … Stone, P. (2022). Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. arXiv preprint arXiv:2203.15041. Google Scholar logo with link to Google Scholar
Khatib, O. (1985). Real-time obstacle avoidance for manipulators and mobile robots. In Proceedings. 1985 ieee international conference on robotics and automation (Vol. 21, p. 500–505). Google Scholar logo with link to Google Scholar
Kim, B., & Pineau, J. (2016). Socially adaptive path planning in human environments using inverse reinforcement learning. International Journal of Social Robotics, 8(1), 51–66. Google Scholar logo with link to Google Scholar
Kitani, K., Ziebart, B., Bagnell, J., & Hebert, M. (2012). Activity forecasting. Computer Vision-ECCV 2012, 201–214. Google Scholar logo with link to Google Scholar
Koren, Y., & Borenstein, J. (1991). Potential field methods and their inherent limitations for mobile robot navigation. In Proceedings. 1991 ieee international conference on robotics and automation (p. 1398–1404 vol.21). Google Scholar logo with link to Google Scholar
Kothari, P., Kreiss, S., & Alahi, A. (2021). Human trajectory forecasting in crowds: A deep learning perspective. IEEE Trans-actions on Intelligent Transportation Systems.Google Scholar logo with link to Google Scholar
Kretzschmar, H., Spies, M., Sprunk, C., & Burgard, W. (2016). Socially compliant mobile robot navigation via inverse reinforcement learning. The International Journal of Robotics Research, 35(11), 1289–1307. Google Scholar logo with link to Google Scholar
Kruse, T., Pandey, A. K., Alami, R., & Kirsch, A. (2013). Human-aware robot navigation: A survey. Robotics and Autonomous Systems, 61(12), 1726–1743. Google Scholar logo with link to Google Scholar
Kuderer, M., Kretzschmar, H., Sprunk, C., & Burgard, W. (2012). Feature-based prediction of trajectories for socially compliant navigation. In Robotics: science and systems.Google Scholar logo with link to Google Scholar
Lam, C.-P., Chou, C.-T., Chang, C.-F., & Fu, L.-C. (2010). Human-centered robot navigation – toward a harmoniously coexisting multi-human and multi-robot environment. In Intelligent robots and systems (iros), 2010 ieee/rsj international conference on (pp. 1813–1818). Taipei, Taiwan.Google Scholar logo with link to Google Scholar
Latombe, J. (1991). Robot motion planning: Edition en anglais. Springer. Retrieved from [URL].
Lerner, A., Chrysanthou, Y., & Lischinski, D. (2007). Crowds by example. In Computer graphics forum (Vol. 261, pp. 655–664). Google Scholar logo with link to Google Scholar
Levine, S., Popovic, Z., & Koltun, V. (2011). Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 241, 19–27.Google Scholar logo with link to Google Scholar
Manso, L. J., Nunez, P., Calderita, L. V., Faria, D. R., & Bachiller, P. (2020). Socnav1: A dataset to benchmark and learn social navigation conventions. Data, 5(1), 7. Google Scholar logo with link to Google Scholar
Martin-Martin, R., Patel, M., Rezatofighi, H., Shenoi, A., Gwak, J., Frankel, E., … Savarese, S. (2021). Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE transactions on pattern analysis and machine intelligence. Google Scholar logo with link to Google Scholar
Mavrogiannis, C., Alves-Oliveira, P., Thomason, W., & Knepper, R. A. (2022). Social momentum: Design and evaluation of a framework for socially competent robot navigation. ACM Transactions on Human-Robot Interaction (THRI), 11(2), 137. Google Scholar logo with link to Google Scholar
Mavrogiannis, C., Baldini, F., Wang, A., Zhao, D., Trautman, P., Steinfeld, A., & Oh, J. (2021). Core challenges of social robot navigation: A survey. arXiv preprint arXiv:2103.05668.Google Scholar logo with link to Google Scholar
Mead, R., Atrash, A., & Matarié, M. J. (2011). Proxemic feature recognition for interactive robots: Automating metrics from the social sciences. In Social robotics (pp. 52–61). Springer. Google Scholar logo with link to Google Scholar
Murphy, R. R. (2019). Introduction to ai robotics. MIT press.Google Scholar logo with link to Google Scholar
Nonaka, S., Inoue, K., Arai, T., & Mae, Y. (2004). Evaluation of human sense of security for coexisting robots using virtual reality. 1st report: evaluation of pick and place motion of hu-manoid robots. In Ieee international conference on robotics and automation, 2004. proceedings. icra’04. 2(004 (Vol. 31, pp. 2770–2775). Google Scholar logo with link to Google Scholar
Nourbakhsh, I., Kunz, C., & Willeke, T. (2003). The mobot museum robot installations: a five year experiment. In Proceedings 2003 ieee/rsj international conference on intelligent robots and systems (iros 2003) (cat. no.03ch37453) (Vol. 41, p. 3636–3641 vol.31). Google Scholar logo with link to Google Scholar
Okal, B., & Arras, K. O. (2016). Formalizing normative robot behavior. In International conference on social robotics (pp. 62–71). Google Scholar logo with link to Google Scholar
Orebäck, A., & Christensen, H. I. (2003). Evaluation of architectures for mobile robotics. Autonomous robots, 14(1), 33–49. Google Scholar logo with link to Google Scholar
Pacchierotti, E., Christensen, H. I., & Jensfelt, P. (2006). Evaluation of passing distance for social robots. In Roman 2006-the 15th ieee international symposium on robot and human interactive communication (pp. 315–320). Google Scholar logo with link to Google Scholar
Pellegrini, S., Ess, A., Schindler, K., & Van Gool, L. (2009). You’ll never walk alone: Modeling social behavior for multi-target tracking. In 2009 ieee 12th international conference on computer vision (pp. 261–268). Google Scholar logo with link to Google Scholar
Pérez-Higueras, N., Caballero, F., & Merino, L. (2018). Learning human-aware path planning with fully convolutional networks. In 2018 ieee international conference on robotics and automation (iera) (pp. 5897–5902). Google Scholar logo with link to Google Scholar
Quinlan, S., & Khatib, O. (1993). Elastic bands: Connecting path planning and control. In [1993] proceedings ieee international conference on robotics and automation (pp. 802–807). Google Scholar logo with link to Google Scholar
Rohmer, E., Singh, S. P. N., & Freese, M. (2013). Cop-peliaSim (formerly V-REP): a Versatile and Scalable Robot Simulation Framework. In Proc. of the international conference on intelligent robots and systems (iros). ([URL])
Rosmann, C., Hoffmann, F., & Bertram, T. (2015). Timed-Elastic-Bands for time-optimal point-to-point nonlinear model predictive control. In 2015 european control conference (ecc) (p. 3352–3357). Google Scholar logo with link to Google Scholar
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of big data, 6(1), 1–48. Google Scholar logo with link to Google Scholar
Stein, P., Spalanzani, A., Santos, V., & Laugier, C. (2016). Leader following: A study on classification and selection. Robotics and Autonomous Systems, 751, 79–95. Google Scholar logo with link to Google Scholar
Syrdal, D. S., Koay, K. L., Walters, M. L., & Dautenhahn, K. (2007). A personalized robot companion-the role of individual differences on spatial preferences in hri scenarios. In Robot and human interactive communication, 2007. ro-man 2007. the 16th ieee international symposium on (pp. 1143–1148). Jeju Island, Korea. Google Scholar logo with link to Google Scholar
Tai, L., Zhang, J., Liu, M., & Burgard, W. (2018). Socially compliant navigation through raw depth inputs with generative adversarial imitation learning. In 2018 ieee international conference on robotics and automation (icra) (pp. 1111–1117). Google Scholar logo with link to Google Scholar
Thrun, S., Beetz, M., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F., … others (2000). Probabilistic algorithms and the interactive museum tour-guide robot minerva. The International Journal of Robotics Research, 19(11), 972–999. Google Scholar logo with link to Google Scholar
Tipaldi, G. D., & Arras, K. O. (2011). Please do not disturb! minimum interference coverage for social robots. In 2011 ieee/rsj international conference on intelligent robots and systems (pp. 1968–1973).Google Scholar logo with link to Google Scholar
Trautman, P., & Krause, A. (2010). Unfreezing the robot: Navigation in dense, interacting crowds. In 2010 ieee/rsj international conference on intelligent robots and systems (pp. 797–803). Google Scholar logo with link to Google Scholar
Tsoi, N., Hussein, M., Espinoza, J., Ruiz, X., & Vázquez, M. (2020). Sean: Social environment for autonomous navigation. In Proceedings of the 8th international conference on human-agent interaction (pp. 281–283). Google Scholar logo with link to Google Scholar
Vadakkepat, P., Tan, K. C., & Ming-Liang, W. (2000). Evolutionary artificial potential fields and their application in real time robot path planning. In Proceedings of the 2000 congress on evolutionary computation. cec 00 (cat. no. 00th8512) (Vol. 11, pp. 256–263). Google Scholar logo with link to Google Scholar
Van den Berg, J., Lin, M., & Manocha, D. (2008). Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 ieee international conference on robotics and automation (pp. 1928–1935). Google Scholar logo with link to Google Scholar
Vasquez, D., Okal, B., & Arras, K. O. (2014). Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In 2014 ieee/rsj international conference on intelligent robots and systems (pp. 1341–1346). Google Scholar logo with link to Google Scholar
Vemula, A., Muelling, K., & Oh, J. (2018). Social attention: Modeling attention in human crowds. In 2018 ieee international conference on robotics and automation (icra) (pp. 4601–4607). Google Scholar logo with link to Google Scholar
Wulfmeier, M., Ondruska, P., & Posner, I. (2015). Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888.Google Scholar logo with link to Google Scholar
Yan, Z., Duckett, T., & Bellotto, N. (2017, September). Online learning for human classification in 3d lidar-based tracking. In In proceedings of the 2017 ieee/rsj international conference on intelligent robots and systems (iros). Vancouver, Canada. Google Scholar logo with link to Google Scholar
Yi, S., Li, H., & Wang, X. (2015). Understanding pedestrian behaviors from stationary crowd groups. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 3488–3496). Google Scholar logo with link to Google Scholar
Zanlungo, F., Ikeda, T., & Kanda, T. (2011). Social force model with explicit collision prediction. EPL(Europhysics Letters), 93(6), 68005. Google Scholar logo with link to Google Scholar
Zhu, Q., Yan, Y., & Xing, Z. (2006). Robot path planning based on artificial potential field approach with simulated annealing. In Sixth international conference on intelligent systems design and applications (Vol. 21, pp. 622–627). Google Scholar logo with link to Google Scholar
Cited by (6)

Cited by six other publications

Xuan, Chenrui, Dongyun Xu, Nan Jiang, Eric Wai Ming Lee & Wei Xie
2026. Unveiling the dynamics and decision-making of dyadic head-on pedestrian collision avoidance: An empirical study. Chaos, Solitons & Fractals 208 ► pp. 118126 ff. DOI logo
Han, James R., Hugues Thomas, Jian Zhang, Nicholas Rhinehart & Timothy D. Barfoot
2025. DR-MPC: Deep Residual Model Predictive Control for Real-World Social Navigation. IEEE Robotics and Automation Letters 10:4 ► pp. 4029 ff. DOI logo
Pekmezci, Mehmet, Emre Ugur & Erhan Oztop
2024. Coupled Conditional Neural Movement Primitives. Neural Computing and Applications 36:30 ► pp. 18999 ff. DOI logo
Yildirim, Yigit & Emre Ugur
2024. Conditional Neural Expert Processes for Learning Movement Primitives From Demonstration. IEEE Robotics and Automation Letters 9:12 ► pp. 10732 ff. DOI logo
Akbulut, Baturhan, Tuba Girgin, Arash Mehrabi, Minoru Asada, Emre Ugur & Erhan Oztop
2023. 2023 IEEE International Conference on Robotics and Automation (ICRA), ► pp. 3904 ff. DOI logo
Roesler, Oliver, Elahe Bagheri, Amir Aly, Silvia Rossi & Rachid Alami
2022. Socially acceptable robot behavior. Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems 23:3 ► pp. 355 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue