Cover not available

Article published In: Interaction Studies
Vol. 25:3 (2024) ► pp.281–312

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Designing and assessing a vocalization-based behavior coding protocol to analyze human-robot interaction in the wild

ORCiD logo with link | Tokyo University of Agriculture and Technology
 | The University of Tokyo
Published online: 27 June 2025
DOI logo with link
https://doi.org/10.1075/is.24004.ind
Abstract
We introduce a vocalization-based behavioral coding protocol, which is designed to assess engagement in in-the-wild child-robot interactions. We evaluate inter-coder agreement between 3–4 coders using the protocol in two training data sets and two experimental data sets in two languages (English and Japanese), both assessing the results as they are, and by grouping behavior codes into broader categories. Using the results of the coding, we analyze segments of the four experimental interactions. We find that this methodology has merit for vocalization-based behavioral analysis, especially when used to build a consensus between multiple behavioral coders to account for ambiguity. It still has several limitations, including a generally low intercoder agreement rate even when the controls are in agreement, which we attribute to the ambiguity of voice recordings of group interactions, meaning that the use of multiple coders to build consensus is not an option but a necessity to eliminate clearly subjective results.
Article outline
  • 1.Introduction
  • 2.Vocalization-based behavior coding protocol for in-the-wild behavioral analysis
    • 2.1Experiments
    • 2.2Selecting the data and preparing the transcript
    • 2.3Broad behavior definitions
    • 2.4Behavior coding protocol
      • Testing data and coders
      • Testing how coders use the similar charts
      • Inter-coder agreement: Analyzing the protocol as-is
      • Converting the protocol categories to the initial nominal categories
      • Inter-coder agreement: After conversion to nominal categories
  • 3.Behavioral analysis
    • 3.1Analysis with nominal categories
      • Japanese data sets
      • English data sets
    • 3.2Analysis with numerical categories
    • 3.3Valence
  • 4.Conclusions
  • References
References (35)
References
Altmann, J. (1973). Observational study of behavior: sampling methods. Behaviour, 49(3):227–267. Google Scholar logo with link to Google Scholar
Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., and Tanaka, F. (2018). Social robots for education: A review. Science Robotics, 3(21):1–9. Google Scholar logo with link to Google Scholar
Boyd, R. L. and Schwartz, H. A. (2021). Natural Language Analysis and the Psychology of Verbal Behavior: The Past, Present, and Future States of the Field. Journal of Language and Social Psychology, 40(1):21–41. Google Scholar logo with link to Google Scholar
Chorney, J. M. L., McMurtry, C. M., Chambers, C. T., and Bakeman, R. (2015). Developing and modifying behavioral coding schemes in pediatric psychology: A practical guide. Journal of Pediatric Psychology, 40(1):154–164. Google Scholar logo with link to Google Scholar
Coronado, E., Indurkhya, X., and Venture, G. (2019). Robots meet children, development of semi-autonomous control systems for children-robot interaction in the wild. In 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM), pages 360–365. Google Scholar logo with link to Google Scholar
Diaz-Boladeras, M., Paillacho, D., Angulo, C., Torres, O., Gonzalez-Dieguez, J., and Albo-Canals, J. (2015). Evaluating Group-Robot Interaction in Crowded Public Spaces: A Week-Long Exploratory Study in the Wild with a Humanoid Robot Guiding Visitors Through a Science Museum. International Journal of Humanoid Robotics, 12(4). Google Scholar logo with link to Google Scholar
Flynn, E. (2008). Investigating children as cultural magnets: Do young children transmit redundant information along diffusion chains? Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1509):3541–3551. Google Scholar logo with link to Google Scholar
Giuliani, M., Mirnig, N., Stollnberger, G., Stadler, S., Buchner, R., and Tscheligi, M. (2015). Systematic analysis of video data from different human-robot interaction studies: a categorization of social signals during error situations. Frontiers in Psychology, 61(July). Google Scholar logo with link to Google Scholar
Guérin, N. A., Gabriels, R. L., Germone, M. M., Sabrina, S. E., Traynor, A., Thomas, K. M., McKenzie, S. J., Slaughter, V., and O Haire, M. E. (2018). Reliability and validity assessment of the Observation of Human-Animal Interaction for Research (OHAIRE) behavior coding tool. Frontiers in Veterinary Science, 5(NOV):1–15. Google Scholar logo with link to Google Scholar
Herbrecht, E., Lazari, O., Notter, M., Schmeck, K., and Spiegel, R. (2019). Process research in early intensive intervention in autism spectrum disorder: Sensitivity to change of the autism behavior coding system. Autism Research, 12(12):1817–1828. Google Scholar logo with link to Google Scholar
Indurkhya, X. and Venture, G. (2024). Lessons in Developing a Behavioral Coding Protocol to Analyze In-the-Wild Child-Robot Events and Experiments. Electronics, 131:1175. Google Scholar logo with link to Google Scholar
Jung, S. E. and Won, E. S. (2018). Systematic review of research trends in robotics education for young children. Sustainability (Switzerland), 10(4):1–24. Google Scholar logo with link to Google Scholar
Kanda, T., Hirano, T., Eaton, D., and Ishiguro, H. (2004). Interactive robots as social partners and peer tutors for children: A field trial. Human-Computer Interaction, 19(1–2):61–84. Google Scholar logo with link to Google Scholar
Komatsubara, T., Shiomi, M., Kanda, T., Ishiguro, H., and Hagita, N. (2014). Can a social robot help children’s understanding of science in classrooms? HAI 2014 — Proceedings of the 2nd International Conference on Human-Agent Interaction, pages 83–90. Google Scholar logo with link to Google Scholar
Krägeloh, C. U., Bharatharaj, J., Kutty, S. K. S., Nirmala, P. R., and Huang, L. (2019). Questionnaires to measure acceptability of social robots: A critical review. Robotics, 8(4):1–14. Google Scholar logo with link to Google Scholar
Lecciso, F., Levante, A., Fabio, R. A., Capr, T., Leo, M., Carcagn, P., Distante, C., Mazzeo, P. L., Spagnolo, P., and Petrocchi, S. (2021). Emotional Expression in Children With ASD: A Pre-Study on a Two-Group Pre-Post-Test Design Comparing Robot-Based and Computer-Based Training. Frontiers in Psychology, 12(July). Google Scholar logo with link to Google Scholar
Leo, M., Carcagn, P., Distante, C., Spagnolo, P., Mazzeo, P. L., Rosato, A. C., Petrocchi, S., Pellegrino, C., Levante, A., Lumè, F. D., and Lecciso, F. (2018). Computational assessment of facial expression production in ASD children. Sensors (Switzerland), 18(11):1–25. Google Scholar logo with link to Google Scholar
Maj, K., Sawicki, K., and Samson, K. (2023). Ready or Not? Examining Acceptance and Fears of Robots in the Labor Market: a Survey of a Polish Sample. Annals of Psychology, 26(4):331–351.Google Scholar logo with link to Google Scholar
Mann, J. (1999). Behavioral sampling methods for cetaceans: a review and a critque. Marine Mammal Science, 15(1):102–122. Google Scholar logo with link to Google Scholar
Manzi, F., Peretti, G., Di Dio, C., Cangelosi, A., Itakura, S., Kanda, T., Ishiguro, H., Massaro, D., and Marchetti, A. (2020). A robot is not worth another: Exploring children’s mental state attribution to different humanoid robots. Frontiers in Psychology, 11(September):1–12. Google Scholar logo with link to Google Scholar
Michaelis, J. E., Cagiltay, B., Ibtasar, R., and Mutlu, B. (2023). “Off script:” Design opportunities emerging from long-term social robot interactions in-the-wild. In ACM/IEEE International Conference on Human-Robot Interaction (HRI ’23), pages 378–387. Google Scholar logo with link to Google Scholar
Nomura, T., Uratani, T., Kanda, T., Matsumoto, K., Kidokoro, H., Suehiro, Y., and Yamada, S. (2015). Why Do Children Abuse Robots? ACM/IEEE International Conference on Human-Robot Interaction, 02–05-Marc:63–64. Google Scholar logo with link to Google Scholar
Pesch, M. H. and Lumeng, J. C. (2017). Methodological considerations for observational coding of eating and feeding behaviors in children and their families. International Journal of Behavioral Nutrition and Physical Activity, 14(1):1–14. Google Scholar logo with link to Google Scholar
Rohlfing, K. J., Altvater-Mackensen, N., Caruana, N., van den Berghe, R., Bruno, B., Tolksdorf, N. F., and Hanulíková, A. (2022). Social/dialogical roles of social robots in supporting children’s learning of language and literacy — A review and analysis of innovative roles. Frontiers in Robotics and AI, 9(October):1–15. Google Scholar logo with link to Google Scholar
Ros, R., Nalin, M., Wood, R., Baxter, P., Looije, R., Demiris, Y., Belpaeme, T., Giusti, A., and Pozzi, C. (2011). Child-robot interaction in the wild: Advice to the aspiring experimenter. ICMI’11 — Proceedings of the 2011 ACM International Conference on Multimodal Interaction, pages 335–342. Google Scholar logo with link to Google Scholar
Serholt, S., Pareto, L., Ekström, S., and Ljungblad, S. (2020). Trouble and Repair in Child-Robot Interaction: A Study of Complex Interactions With a Robot Tutee in a Primary School Classroom. Frontiers in Robotics and AI, 7(April):1–13. Google Scholar logo with link to Google Scholar
Sienkiewicz, B., Sejnova, G., Gajewski, P., Vavrecka, M., and Indurkhya, B. (2023). How language of interaction affects the user perception of a robot. In Ali, A. A., editor, Proceedings of the International Conference on Social Robotics (ICSR2023), Dec 3–7, Doha (Qatar), pages 308–321, Singapore. Springer Nature.Google Scholar logo with link to Google Scholar
Singh, D. K., Kumar, M., Fosch-Villaronga, E., Singh, D., and Shukla, J. (2023). Ethical Considerations from Child-Robot Interactions in Under-Resourced Communities. International Journal of Social Robotics, 15(12):2055–2071. Google Scholar logo with link to Google Scholar
Tanaka, F., Movellan, J. R., Fortenberry, B., and Aisaka, K. (2006). Daily HRI evaluation at a classroom environment: reports from dance interaction experiments. Proc 1st Annual Conf on HumanRobot Interaction HRI, pages 3–9. Google Scholar logo with link to Google Scholar
Tobis, S., Neumann-Podczaska, A., Kropinska, S., and Suwalska, A. (2021). Unraq — a questionnaire for the use of a social robot in care for older persons. A multi-stakeholder study and psychometric properties. International Journal of Environmental Research and Public Health, 18(11). Google Scholar logo with link to Google Scholar
Türker, B. B., Buçinca, Z., Erzin, E., Yemez, Y., and Sezgin, M. (2017). Analysis of engagement and user experience with a laughter responsive social robot. Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, 2017-Augus:844–848. Google Scholar logo with link to Google Scholar
Whiten, A., McGuigan, N., Marshall-Pescini, S., and Hopper, L. M. (2009). Emulation, imitation, over-imitation and the scope of culture for child and chimpanzee. Philosophical Transactions of the Royal Society B: Biological Sciences, 364(1528):2417–2428. Google Scholar logo with link to Google Scholar
Wood, L. J., Dautenhahn, K., Rainer, A., Robins, B., Lehmann, H., and Syrdal, D. S. (2013). Robot-Mediated Interviews — How Effective Is a Humanoid Robot as a Tool for Interviewing Young Children? PLoS ONE, 8(3). Google Scholar logo with link to Google Scholar
Wróbel, A., Źróbek, K., Indurkhya, B., Schaper, M. M., Gunia, A., and Zguda, P. M. (2023a). Are robots vegan? Unexpected behaviours in child-robot interactions and their design implications. In CHI EA ’23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing, Systems April 23–28, Hamburg, Germany, pages 1–7. Google Scholar logo with link to Google Scholar
Wróbel, A., Źróbek, K., Schaper, M.-M., Zguda, P., and Indurkhya, B. (2023b). Age-Appropriate Robot Design: In-The-Wild Child-Robot Interaction Studies of Perseverance Styles and Robot’s Unexpected Behavior. In Proceedings of the 32nd IEEE International Conference on Robot & Human Interactive Communication (Ro-Man 2023), Aug 28–31, Busan, S. Korea, pages 1451–1458. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue