In:Gaze in Human-Robot Communication
Edited by Frank Broz, Hagen Lehmann, Bilge Mutlu and Yukiko Nakano
[Benjamins Current Topics 81] 2015
► pp. 47–70
Interactions between a quiz robot and multiple participants
Focusing on speech, gaze and bodily conduct in Japanese and English speakers
Published online: 16 December 2015
https://doi.org/10.1075/bct.81.04yam
https://doi.org/10.1075/bct.81.04yam
This paper reports on a quiz robot experiment in which we explore similarities and differences in human participant speech, gaze, and bodily conduct in responding to a robot’s speech, gaze, and bodily conduct across two languages. Our experiment involved three-person groups of Japanese and English-speaking participants who stood facing the robot and a projection screen that displayed pictures related to the robot’s questions. The robot was programmed so that its speech was coordinated with its gaze, body position, and gestures in relation to transition relevance places (TRPs), key words, and deictic words and expressions (e.g. this, this picture) in both languages. Contrary to findings on human interaction, we found that the frequency of English speakers’ head nodding was higher than that of Japanese speakers in human-robot interaction (HRI). Our findings suggest that the coordination of the robot’s verbal and non-verbal actions surrounding TRPs, key words, and deictic words and expressions is important for facilitating HRI irrespective of participants’ native language.
References (38)
Bavelas, J.B., Coates, L., & Johnson, T. (2002). Listener responses as a collaborative process: The role of gaze. Journal of Communication, 52(3), 566–580.
CMU cross-cultural receptionist corpus. (2012). Retrieved from [URL]
Diginfonews (2010). Museum Guide Robot: Diginfo [Video file]. Retrieved from [URL]
Face API: Retrieved September 6, 2013. from [URL]
Fox, B.A., Hayashi, M., & Jasperson, R. (1996). Resources and repair: A cross-linguistic study of syntax and repair. Studies in Interactional Sociolinguistics, 13, 185–237.
Goodwin, C. (1981). Conversational organization: Interaction between speakers and hearers. Language, thought, and culture. New York: Academic Press.
. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522.
Goodwin, M.H. (1990). He-said-she-said: Talk as social organization among black children (Vol. 618). Bloomington: Indiana University Press.
Hayashi, M. (2010). An overview of the question-response system in Japanese. Journal of Pragmatics, 42(10), 2685–2702.
Iwasaki, S. (2009). Initiating interactive turn spaces in Japanese conversation: Local projection and collaborative action. Discourse Processes, 46(2–3), 226–246.
Jefferson, G. (1984). Transcript notation. In J.M. Atkinson & J. Hertiage (Eds.), Structures of social action. Studies in conversation analysis (pp. ix–xvi). Cambridge, UK: Cambridge University Press.
Kendon, A. (1967). Some functions of gaze-direction in social interaction. Acta psychologica, 26, 22–63.
Knight, H., & Simmons, R. (2012). Estimating human interest and attention via gaze analysis. In
Proceedings of the 12th International Conference on Intelligent Virtual Agents
(IVA’12, pp. 245–251).
Kogure, M. (2007). Nodding and smiling in silence during the loop sequence of backchannels in Japanese conversation. Journal of Pragmatics, 39(7), 1275–1289.
Lee, C., Lesh, N., Sidner, C.L., Morency, L.P., Kapoor, A., & Darrell, T. (2004). Nodding in conversations with a robot. In
CHI’04 Extended Abstracts on Human Factors in Computing Systems
(pp. 785–786). ACM.
Makatchev, M., Simmons, R., & Sakr, M. (2012). A cross-cultural corpus of annotated verbal and nonverbal behaviors in receptionist encounters. In
Gaze in HRI: From Modeling to Communication workshop at 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI)
, Boston, USA, March, 2012.
Maynard, S.K. (1990). Conversation management in contrast: Listener response in Japanese and American English. Journal of Pragmatics, 14(3), 397–412.
Mutlu, B., Shiwa, T., Kanda, T., Ishiguro, H., & Hagita, N. (2009). Footing in human-robot conversations: how robots might shape participant roles using gaze cues. In
Proceedings of the 4th ACM/IEEE International Conference on Human-Robot Interaction
(pp. 61–68). ACM.
Nakano, Y., & Rehm, M. (2009). Multimodal corpus analysis as a method for ensuring cultural usability of embodied conversational agents. In Human centered design (pp. 521–530). Springer Berlin: Heidelberg.
Nomura, T., Kanda, T., Suzuki, T., Han, J., Shin, N., Burke, J., & Kato, K. (2007). Implications on humanoid robots in pedagogical applications from cross-cultural analysis between Japan, Korea, and the USA. In
Robot and Human Interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on
(pp. 1052–1057). IEEE.
Pitsch, K., Vollmer, A.L., & Muhlig, M. (2013). Robot feedback shapes the tutor’s presentation. How a robot’s online gaze strategies lead to micro-adaptation of the human’s conduct. Interaction Studies, 14(2), 268–296.
Rossano, F. (2013). Gaze in conversation. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 308–329). Malden, MA: Blackwell.
Rossano, F., Brown, P., & Levinson, S.C. (2009). Gaze, questioning and culture. In J. Sidnell (Ed.), Conversation analysis: Comparative perspectives (pp. 187–249). Cambridge, UK: Cambridge University Press.
Sacks, H., Schegloff, E.A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), Part 1, 696–735.
Sacks, H. (1987). On the preferences for agreement and contiguity in sequences in conversation. In G. Button & J.R.E. Lee (Eds.), Talk and social organization (pp. 54–69). Clevedon, UK: Multilingual Matters.
Schegloff, E.A. (2007). Sequence organization in interaction: A primer in conversation analysis (Vol. 1). Cambridge University Press.
Suchman, L. (2006). Human-machine reconfigurations: Plans and situated actions. Cambridge, UK: Cambridge University Press.
Stivers, T. (2008). Stance, alignment, and affiliation during storytelling: When nodding is a token of affiliation. Research on Language and Social Interaction, 41(1), 31–57.
. (2010). An overview of the question-response system in American English conversation. Journal of Pragmatics, 42(10), 2772–2781.
Stivers, T., & Rossano, F. (2010). Mobilizing response. Research on Language and Social Interaction, 43(1), 3–31.
Tanaka, H. (1999). Turn-taking in Japanese conversation: A study in grammar and interaction (Vol. 3). Amsterdam: John Benjamins.
Traum, D., Aggarwal, P., Artstein, R., Foutz, S., Gerten, J., Katsamanis, A., Leuski, A., Noren, D., & Swartout, W. (2012). Ada and grace: Direct interaction with museum visitors. In Y. Nakano et al. (Eds.), Proceedings of the 12th International Conference on Intelligent Virtual Agents (IVA’12, pp. 245–251). Heidelberg: Springer.
Yamazaki, A., Yamazaki, K., Burdelski, M., Kuno, Y., & Fukushima, M. (2010). Coordination of verbal and non-verbal actions in human-robot interaction at museums and exhibitions. Journal of Pragmatics, 42(9), 2398–2414.
Yamazaki, A., Yamazaki, K., Kuno, Y., Burdelski, M., Kawashima, M., & Kuzuoka, H. (2008). Precision timing in human-robot interaction: Coordination of head movement and utterance. In
Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems
(CHI’ 08, pp. 131–140). ACM.
Yamazaki, A., Yamazaki, K., Ohyama, T., Kobayashi, Y., & Kuno, Y. (2012). A techno-sociological solution for designing a museum guide robot: Regarding choosing an appropriate visitor. In
Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction
(HRI’ 12, pp. 309–316). ACM.
Yamazaki, K., Yamazaki, A., Okada, M., Kuno, Y., Kobayashi, Y., Hoshi, Y., & Heath, C. (2009). Revealing gauguin: Engaging visitors in robot guide’s explanation in an art museum. In
Proceedings of the 27th International Conference on Human Factors in Computing Systems
(CHI’ 09, pp. 1437–1446). ACM.