Cover not available

Article published In: Interaction Studies
Vol. 22:1 (2021) ► pp.55–80

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

What’s to bullying a bot?

Correlates between chatbot humanlikeness and abuse

 | University of Canterbury
 | University of Canterbury
 | Bielefeld University
Published online: 17 September 2021
DOI logo with link
https://doi.org/10.1075/is.20002.kei
Abstract
In human-chatbot interaction, users casually and regularly offend and abuse the chatbot they are interacting with. The current paper explores the relationship between chatbot humanlikeness on the one hand and sexual advances and verbal aggression by the user on the other hand. 283 conversations between the Cleverbot chatbot and its users were harvested and analysed. Our results showed higher counts of user verbal aggression and sexual comments towards Cleverbot when Cleverbot appeared more humanlike in its behaviour. Caution is warranted with the interpretation of the results however as no experimental manipulation was conducted and causality can thus not be inferred. Nonetheless, the findings are relevant for both the research on the abuse of conversational agents, and the development of efficient approaches to discourage or prevent verbal aggression by chatbot users.
Article outline
  • 1.Introduction
    • 1.1Literature
      • 1.1.1Incidence: Nonhuman conversation agents suffer more abuse than humans
      • 1.1.2Ethical implications
      • 1.1.3Need for a theoretical framework of chatbot abuse
    • 1.2Current study
  • 2.Method
    • 2.1Procedure
    • 2.2Dataset
      • 2.2.1Data collection
    • 2.3Coding procedure
    • 2.4Variables
      • 2.4.1Coded variables
        • Verbal aggression (Offence)
        • Sexual references (SexRef)
        • Nonsensical reply by Cleverbot (Nonsense)
        • Insult by Cleverbot (Insult)
        • Claiming to be (not) human (Humanity/Non-human user)
        • Gender
      • 2.4.2Other variables
        • Turn count
        • Approximate Loebner test (Loebner)
  • 3.Results
    • 3.1Preliminary analyses
      • 3.1.1Conversation descriptives
      • 3.1.2Agreement
    • 3.2Main analyses
      • 3.2.1Verbal aggression
      • 3.2.2Sexual comments
  • 4.Discussion
    • 4.1Limitations
    • 4.2Conclusion
  • Acknowledgements
  • Notes
  • References
References (57)
References
Ardissono, L., Boella, G., and Lesmo, L. (2000). A plan-based agent architecture for interpreting natural language dialogue. International Journal of Human-Computer Studies, 52(4):583–635. Google Scholar logo with link to Google Scholar
Bartneck, C., Reichenbach, J., and Carpenter, J. (2008). The carrot and the stick – the role of praise and punishment in human-robot interaction. Interaction Studies – Social Behaviour and Communication in Biological and Artificial Systems, 9(2):179–203. Google Scholar logo with link to Google Scholar
Bozdogan, H. (1987). Model selection and akaike’s information criterion (aic): The general theory and its analytical extensions. Psychometrika, 52(3):345–370. Google Scholar logo with link to Google Scholar
Brahnam, S. (2005). Strategies for handling customer abuse of ecas. Abuse: The darker side of humancomputer interaction, pages 62–67.Google Scholar logo with link to Google Scholar
Brahnam, S. and De Angeli, A. (2012). Gender affordances of conversational agents. Interacting with Computers, 24(3):139–153. Google Scholar logo with link to Google Scholar
Brscić, D., Kidokoro, H., Suehiro, Y., and Kanda, T. (2015). Escaping from children’s abuse of social robots. In Proceedings of the International Conference on Human-Robot Interaction, pages 59–66, Portland, USA. ACM/IEEE. Google Scholar logo with link to Google Scholar
Burnham, K. P. and Anderson, D. R. (2003). Model selection and multimodel inference: a practical information-theoretic approach. Springer Science & Business Media, New York.Google Scholar logo with link to Google Scholar
Chin, H. and Yi, M. Y. (2019). Should an agent be ignoring it?: A study of verbal abuse types and conversational agents’ response styles. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–6. ACM. Google Scholar logo with link to Google Scholar
Connolly, J. (2020). Preventing robot abuse through emotional robot responses. In Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pages 558–560. Google Scholar logo with link to Google Scholar
Cowie, H. and Berdondini, L. (2002). The expression of emotion in response to bullying. Emotional and Behavioural Difficulties, 7(4):207–214. Google Scholar logo with link to Google Scholar
Curry, A. C. and Rieser, V. (2018). # metoo alexa: How conversational systems respond to sexual harassment. In Proceedings of the Second ACL Workshop on Ethics in Natural Language Processing, pages 7–14. Google Scholar logo with link to Google Scholar
Darling, K. (2012). Extending legal rights to social robots. In We Robot Conference, University of Miami, pages 1–24, Miami, USA. University of Miami. Google Scholar logo with link to Google Scholar
De Angeli, A. (2009). Ethical implications of verbal disinhibition with conversational agents. Psych-Nology Journal, 7(1):49–57.Google Scholar logo with link to Google Scholar
De Angeli, A. and Brahnam, S. (2006). Sex stereotypes and conversational agents. In Gender and Interaction: real and virtual women in a male world. Venice, Italy, pages 1–4.Google Scholar logo with link to Google Scholar
(2008). I hate you! disinhibition with virtual partners. Interacting with computers, 20(3):302–310. Google Scholar logo with link to Google Scholar
De Angeli, A., Brahnam, S., Wallis, P., and Dix, A. (2006). Misuse and abuse of interactive technologies. In CHI’06 Extended Abstracts on Human Factors in Computing Systems, pages 1647–1650, Montreal, Canada. ACM. Google Scholar logo with link to Google Scholar
De Angeli, A. and Carpenter, R. (2005). Stupid computer! abuse and social identities. In Proceedings of Abuse: The dark side of human-computer interaction, An INTERACT 2005 workshop, pages 19–25.Google Scholar logo with link to Google Scholar
De Angeli, A., Johnson, G. I., and Coventry, L. (2001). The unfriendly user: exploring social reactions to chatterbots. In Proceedings of The International Conference on Affective Human Factors Design, London, pages 467–474.Google Scholar logo with link to Google Scholar
De Swert, K. (2012). Calculating inter-coder reliability in media content analysis using Krippendorffs Alpha. Center for Politics and Communication, University of Amsterdam, the Netherlands.Google Scholar logo with link to Google Scholar
Dindia, K., Fitzpatrick, M. A., and Kenny, D. A. (1997). Self-disclosure in spouse and stranger interaction: A social relations analysis. Human Communication Research, 23(3):388–412. Google Scholar logo with link to Google Scholar
Fessler, L. (2017a). Apple and amazon are under fire for siri and alexas responses to sexual harassment. [URL]
(2017b). We tested bots like siri and alexa to see who would stand up to sexual harassment. [URL]
Haslam, N. (2006). Dehumanization: An integrative review. Personality and social psychology review, 10(3):252–264. Google Scholar logo with link to Google Scholar
Haslam, N., Loughnan, S., Kashima, Y., and Bain, P. (2008). Attributing and denying humanness to others. European review of social psychology, 19(1):55–85. Google Scholar logo with link to Google Scholar
Hern, A. (2010). Apple made siri deflect questions on feminism, leaked papers reveal. [URL]. [Online; recovered 23 June 2020].
Hill, J., Ford, W. R., and Farreras, I. G. (2015). Real conversations with artificial intelligence: A comparison between human-human online conversations and human-chatbot conversations. Computers in Human Behavior, 491:245–250. Google Scholar logo with link to Google Scholar
Ho, A., Hancock, J., and Miner, A. S. (2018). Psychological, relational, and emotional effects of self-disclosure after conversations with a chatbot. Journal of Communication, 68(4):712–733. Google Scholar logo with link to Google Scholar
Hutchinson, M. K. and Holtman, M. C. (2005). Analysis of count data using poisson regression. Research in Nursing & Health, 28(5):408–418. Google Scholar logo with link to Google Scholar
Jay, T. (2009). The utility and ubiquity of taboo words. Perspectives on Psychological Science, 4(2):153–161. Google Scholar logo with link to Google Scholar
Katsyri, J., Forger, K., Mäkäräinen, M., and Takala, T. (2015). A review of empirical evidence on different uncanny valley hypotheses: Support for perceptual mismatch as one road to the valley of eeriness. Frontiers in psychology, 61. Google Scholar logo with link to Google Scholar
Keijsers, M. and Bartneck, C. (2018). Mindless robots get bullied. In Proceedings of the International Conference on Human-Robot Interaction, pages 205–214, New York, USA. ACM/IEEE. Google Scholar logo with link to Google Scholar
Keijsers, M., Bartneck, C., and Kazmi, H. S. (2019a). Cloud-based sentiment analysis for interactive agents. In Proceedings of the 7th International Conference on Human-Agent Interaction, pages 43–50. Google Scholar logo with link to Google Scholar
(2019b). Cloud-based sentiment analysis for interactive agents. In Proceedings of the 7th International Conference on Human-Agent Interaction (HAI), pages 43–50. Google Scholar logo with link to Google Scholar
Krach, S., Hegel, F., Wrede, B., Sagerer, G., Binkofski, F., and Kircher, T. (2008). Can machines think? interaction and perspective taking with robots investigated via fmri. PloS One, 3(7):e2597. Google Scholar logo with link to Google Scholar
Lee, M. K., Kiesler, S., and Forlizzi, J. (2010). Receptionist or information kiosk: how do people talk with a robot? In Proceedings of the 2010 ACM conference on Computer supported cooperative work, pages 31–40. Google Scholar logo with link to Google Scholar
Lortie, C. L. and Guitton, M. J. (2011). Judgment of the humanness of an interlocutor is in the eye of the beholder. PLoS One, 6(9):e25085. Google Scholar logo with link to Google Scholar
Lowry, P. B., Zhang, J., Wang, C., and Siponen, M. (2016). Why do adults engage in cyberbullying on social media? an integration of online disinhibition and deindividuation effects with the social structure and social learning model. Information Systems Research, 27(4):962–986. Google Scholar logo with link to Google Scholar
MacDorman, K. F. and Chattopadhyay, D. (2016). Reducing consistency in human realism increases the uncanny valley effect; increasing category uncertainty does not. Cognition, 1461:190–205. Google Scholar logo with link to Google Scholar
Mauldin, M. L. (1994). Chatterbots, tinymuds, and the turing test: Entering the loebner prize competition. In AAAI, volume 941, pages 16–21.Google Scholar logo with link to Google Scholar
Moore, S. (2018). Gartner says 25 percent of customer service operations will use virtual customer assistants by 2020.Google Scholar logo with link to Google Scholar
Mori, M. et al. (1970). The uncanny valley. Energy, 7(4):33–35.Google Scholar logo with link to Google Scholar
Nass, C., Steuer, J., and Tauber, E. R. (1994). Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 72–78, Boston, USA. ACM.Google Scholar logo with link to Google Scholar
Nomura, T., Kanda, T., Kidokoro, H., Suehiro, Y., and Yamada, S. (2017). Why do children abuse robots? Interaction Studies, 17(3):347–369. Google Scholar logo with link to Google Scholar
Oberman, L. M., McCleery, J. P., Ramachandran, V. S., and Pineda, J. A. (2007). Eeg evidence for mirror neuron activity during the observation of human and robot actions: Toward an analysis of the human qualities of interactive robots. Neurocomputing, 70(13–15):2194–2203. Google Scholar logo with link to Google Scholar
Paetzel, M., Peters, C., Nyström, I., and Castellano, G. (2016). Congruency matters – how ambiguous gender cues increase a robots uncanniness. In International Conference on Social Robotics, pages 402–412. Springer. Google Scholar logo with link to Google Scholar
Pennebaker, J. W., Booth, R. J., Boyd, R. L., and Francis, M. E. (2015a). Linguistic inquiry and word count: Liwc 2015 [computer software]. pennebaker conglomerates.Google Scholar logo with link to Google Scholar
Pennebaker, J. W., Boyd, R. L., Jordan, K., and Blackburn, K. (2015b). The development and psychometric properties of liwc2015. Technical report, The University of Texas at Austin.Google Scholar logo with link to Google Scholar
Reeves, B. and Nass, C. (1996). The Media Equation. CSLI Publications and Cambridge University Press, Cambridge.Google Scholar logo with link to Google Scholar
Rehm, M. and Krogsager, A. (2013). Negative affect in human robot interaction – impoliteness in unexpected encounters with robots. In Proceedings of the 22nd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 45–50. IEEE. Google Scholar logo with link to Google Scholar
Slater, M., Antley, A., Davison, A., Swapp, D., Guger, C., Barker, C., Pistrang, N., and Sanchez-Vives, M. V. (2006). A virtual reprise of the stanley milgram obedience experiments. PloS one, 1(1):e39. Google Scholar logo with link to Google Scholar
Sokol, N., Bussey, K., and Rapee, R. M. (2016). Victims responses to bullying: The gap between students evaluations and reported responses. School Mental Health, 8(4):461–475. Google Scholar logo with link to Google Scholar
Strait, M., Contreras, V., and Vela, C. D. (2018). Verbal disinhibition towards robots is associated with general antisociality. arXiv e-prints.Google Scholar logo with link to Google Scholar
Suler, J. (2004). The online disinhibition effect. Cyberpsychology & behavior, 7(3):321–326. Google Scholar logo with link to Google Scholar
Tan, X. Z., Vázquez, M., Carter, E. J., Morales, C. G., and Steinfeld, A. (2018). Inducing bystander interventions during robot abuse with social mechanisms. In Proceedings of the International Conference on Human-Robot Interaction, pages 169–177, New York, USA. ACM/IEEE. Google Scholar logo with link to Google Scholar
Veletsianos, G., Scharber, C., and Doering, A. (2008). When sex, drugs, and violence enter the classroom: Conversations between adolescents and a female pedagogical agent. Interacting with computers, 20(3):292–301. Google Scholar logo with link to Google Scholar
Whitby, B. (2008). Sometimes its hard to be a robot: A call for action on the ethics of abusing artificial agents. Interacting with Computers, 20(3):326–333. Google Scholar logo with link to Google Scholar
Zhang, Z. (2016). Variable selection with stepwise and best subset approaches. Annals of Translational Medicine, 4(7). Google Scholar logo with link to Google Scholar
Cited by (21)

Cited by 21 other publications

Cao, Cuicui, Yingying Hu & Qianfan Zhang
2026. “Hey Siri, You’re Dumb!”: The Fit Effect of Power Experience and Desire for Power on User Aggression Toward Conversational Agents. International Journal of Human–Computer Interaction 42:3 ► pp. 1490 ff. DOI logo
Dobrosovestnova, Anna, David J. Bailey, Ralf Vetter & Masoumeh Mansouri
2026. Proceedings of the 21st ACM/IEEE International Conference on Human-Robot Interaction, ► pp. 1322 ff. DOI logo
Giebeler, Yasmina, Basil Wahn & Eva Wiese
2026. Wall-E vs. Terminator: The relationship between physical appearance and dimensions of mind perception. Computers in Human Behavior: Artificial Humans 7 ► pp. 100271 ff. DOI logo
Lancaster, Karen
2026. Ethical Chatbot Creation of Real People. In Crisis or Redemption with AI and Robotics? The Dawn of a New Era [Lecture Notes in Networks and Systems, 1556], ► pp. 3 ff. DOI logo
Martinez, Julien
2026. AI at a Turning Point: Could It Become a Driving Force in the Prevention and Treatment of Pedophilia?. Journal of Forensic Nursing DOI logo
Alqurashi, Turki S.
2025. The Impact of Artificial Intelligence on Intimate Partner Relationships. The American Journal of Family Therapy ► pp. 1 ff. DOI logo
Chen, Yuling, Jiacheng Cao, Shuping Liu, Mei Huang, Fang Wan & Charles Weizheng Chen
2025. Friend or Foe? An empirical study on the mechanisms triggering robot abuse. Journal of Hospitality and Tourism Management 64 ► pp. 101318 ff. DOI logo
Djufril, Ray, Jessica R. Frampton & Silvia Knobloch-Westerwick
2025. Love, marriage, pregnancy: Commitment processes in romantic relationships with AI chatbots. Computers in Human Behavior: Artificial Humans 4 ► pp. 100155 ff. DOI logo
Guan, Biyu, Xin Li, Zhenshuo Luo & Pei Liu
2025. Can (A)I Arouse You? The Impact of AI Services on Consumer Pro-Environmental Behavior. Journal of Hospitality & Tourism Research 49:5 ► pp. 932 ff. DOI logo
Li, Jingshu, Zicheng Zhu, Renwen Zhang & Yi-Chieh Lee
2025. Exploring the Effects of Chatbot Anthropomorphism and Human Empathy on Human Prosocial Behavior Toward Chatbots. Proceedings of the ACM on Human-Computer Interaction 9:7 ► pp. 1 ff. DOI logo
Lima, Eleonora & Tiffany Morisseau
2025. Can chatbots teach us how to behave? Examining assumptions about user interactions with AI assistants and their social implications. Frontiers in Artificial Intelligence 8 DOI logo
Rafikova, Antonina & Anatoly Voronin
2025. Human–chatbot communication: a systematic review of psychological studies. AI & SOCIETY 40:7 ► pp. 5389 ff. DOI logo
Rezzani, Andrea, María Menéndez-Blanco, Brad J. Bushman & Antonella De Angeli
2025. What is robot abuse? A sociotechnical definition. Behaviour & Information Technology ► pp. 1 ff. DOI logo
Ungless, Eddie L., Björn Ross & Zachary Horne
2025. Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff. DOI logo
Wang, Cheng-Yen
2025. Exploring the Dark Tetrad in Human–GenAI Relationships: A Multi-Source Evaluation of GenAI Abuse. Social Science Computer Review DOI logo
Yamada, Sachie, Takayuki Kanda & Kohki Arimitsu
2025. Development and Validation of Moral Concern for Robot Scale (MCRS). ACM Transactions on Human-Robot Interaction 14:4 ► pp. 1 ff. DOI logo
De Cicco, Roberta
2024. Exploring the Dark Corners of Human-Chatbot Interactions: A Literature Review on Conversational Agent Abuse. In Chatbot Research and Design [Lecture Notes in Computer Science, 14524], ► pp. 185 ff. DOI logo
Gille, Felix, Kimon Papadopoulos, Jana Sedlakova, Federica Zavattaro & Caroline Brall
2024. Vertrauen. In Digitalwirtschaft, ► pp. 129 ff. DOI logo
Ladak, Ali, Jamie Harris & Jacy Reese Anthis
2024. Proceedings of the CHI Conference on Human Factors in Computing Systems , ► pp. 1 ff. DOI logo
Xi, Yipeng, Aitong Ji & Weihua Yu
2024. Enhancing or impeding? Exploring the dual impact of anthropomorphism in large language models on user aggression. Telematics and Informatics 95 ► pp. 102194 ff. DOI logo
Jones, Mirabelle, Christina Neumayer & Irina Shklovski
2023. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue