Cover not available

Article published In: Interaction Studies
Vol. 19:3 (2018) ► pp.487–498

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Frustration in the face of the driver

A simulator study on facial muscle activity during frustrated driving

 | Institute of Transportation Systems, German Aerospace Center
 | Institute of Transportation Systems, German Aerospace Center
 | Institute of Flight Guidance, German Aerospace Center
 | Institute of Transportation Systems, German Aerospace Center
Published online: 13 March 2019
DOI logo with link
https://doi.org/10.1075/is.17005.ihm
Abstract
Frustration in traffic is one of the causes of aggressive driving. Knowledge whether a driver is frustrated may be utilized by future advanced driver assistance systems to counteract this source of crashes. One possibility to achieve this is to automatically recognize facial expressions of drivers. However, only little is known about the facial expressions of frustrated drivers. Here, we report the results of a driving simulator study investigating the facial muscle activity that comes along with frustration. Twenty-eight participants were video-taped during frustrated and non-frustrated driving situations. Their facial muscle activity was manually coded according to the Facial Action Coding System. Participants showed significantly more facial muscle activity in the mouth region. Thus, recording facial muscle behavior potentially provides traffic researchers and assistance system developers with the possibility to recognize frustration while driving.
Article outline
  • Methods
    • Participants
    • Experimental set-up
    • Experimental design and cover story
    • Subjective ratings
    • Video coding
  • Results
    • Ratings
    • Activity in facial AUs
  • Discussion
  • Acknowledgements
  • References
References (24)
References
Barrett, L. F. (2016). The theory of constructed emotion: an active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, nsw154. Google Scholar logo with link to Google Scholar
Bruce, V. (1992). What the human face tells the human mind: some challenges for the robot-human interface. In Proceedings of IEEE International Workshop on Robot and Human Communication (pp. 44–51). Google Scholar logo with link to Google Scholar
Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the Self-Assessment Manikin and the Semantic Differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. Google Scholar logo with link to Google Scholar
D’Mello, S., Craig, S., Gholson, B., Franklin, S., Picard, R., & Graesser, A. (2005). Integrating affect sensors in an intelligent tutoring system. In Affective Interactions: The Computer in the Affective Loop Workshop at 2005 International Conference on Intelligent User Interfaces. New York: AMC Press.Google Scholar logo with link to Google Scholar
Deffenbacher, J. L., Lynch, R. S., Oetting, E. R., & Swaim, R. C. (2002). The Driving Anger Expression Inventory: a measure of how people express their anger on the road. Behaviour Research and Therapy, 40(6), 717–737. Google Scholar logo with link to Google Scholar
Donkor, R., Burnett, G., & Sharples, S. (2014). Measuring the emotional validity of driving simulators. Advances in Transportation Studies, (Special, Issue Special Vol1), 51–64.Google Scholar logo with link to Google Scholar
Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3), 169–200. Google Scholar logo with link to Google Scholar
Ekman, P., & Friesen, W. V. (2003). Unmasking the face: A guide to recognizing emotions from facial clues. Cambridge, MA: Malor Books.Google Scholar logo with link to Google Scholar
Ekman, P., Friesen, W. V., & Hager, J. (2002). The Investigator’s Guide for the Facial Action Coding System. Salt Lake City: A Human face.Google Scholar logo with link to Google Scholar
Gao, H., Yuce, A., & Thiran, J.-P. (2014). Detecting emotional stress from facial expressions for driving safety. In IEEE International Conference on Image Processing (ICIP) (pp. 5961–5965). Google Scholar logo with link to Google Scholar
Gehrig, T., & Ekenel, H. K. (2011). A common framework for real-time emotion recognition and facial action unit detection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops) (pp. 1–6).Google Scholar logo with link to Google Scholar
Gosselin, P., Perron, M., & Beaupré, M. (2010). The voluntary control of facial action units in adults. Emotion, 10(2), 266–271. Google Scholar logo with link to Google Scholar
Grafsgaard, J. F., Wiggins, J. B., Boyer, K. E., Wiebe, E. N., & Lester, J. C. (2013). Automatically recognizing facial indicators of frustration: A learning-centric analysis. In Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 159–165). Google Scholar logo with link to Google Scholar
Hamm, J., Kohler, C. G., Gur, R. C., & Verma, R. (2011). Automated Facial Action Coding System for dynamic analysis of facial expressions in neuropsychiatric disorders. Journal of Neuroscience Methods, 200(2), 237–256. Google Scholar logo with link to Google Scholar
Hart, S. G., & Staveland, L. (1988). Development of the NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology, 521, 139–183. Google Scholar logo with link to Google Scholar
Healey, J., & Picard, R. (2005). Detecting stress during real-world driving tasks using physiological sensors. IEEE Transactions on Intelligent Transportation Systems, 6(2), 156–166. Google Scholar logo with link to Google Scholar
Hoque, M. E., McDuff, D. J., & Picard, R. W. (2012). Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Transactions on Affective Computing, 3(3), 323–334. Google Scholar logo with link to Google Scholar
Lazarus, R. S. (1991). Progress on a cognitive-motivational-relational theory of emotion. American Psychologist, 46(8), 819–834. Google Scholar logo with link to Google Scholar
Lee, Y.-C. (2010). Measuring drivers’ frustration in a driving simulator. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(19), 1531–1535. Google Scholar logo with link to Google Scholar
Lee, Y.-C., & LaVoie, N. (2014). Relationship between frustration justification and vehicle control behaviors ? A simulator study. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 58(1), 2235–2239. Google Scholar logo with link to Google Scholar
Malta, L., Miyajima, C., Kitaoka, N., & Takeda, K. (2011). Analysis of real-world driver’s frustration. IEEE Transactions on Intelligent Transportation Systems, 12(1), 109–118. Google Scholar logo with link to Google Scholar
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. Google Scholar logo with link to Google Scholar
Scherer, K. R. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729. Google Scholar logo with link to Google Scholar
Tews, T.-K., Oehl, M., Siebert, F. W., Höger, R., & Faasch, H. (2011). Emotional human-machine interaction: cues from facial expressions. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, … (Eds.), Lecture Notes in Computer Science. Human Interface and the Management of Information. Interacting with Information (Vol. 67711, pp. 641–650). Berlin: Springer. Google Scholar logo with link to Google Scholar
Cited by (26)

Cited by 26 other publications

Seaman, Sean, Peihan Zhong, Linda Angell, Joshua Domeyer & John Lenneman
2025. Identifying Negative Driver States that Share Commonalities for Interventions. SAE International Journal of Advances and Current Practices in Mobility 07:5 ► pp. 2189 ff. DOI logo
Karas, Vincent, Dagmar M. Schuller & Björn W. Schuller
2024. Audiovisual Affect Recognition for Autonomous Vehicles: Applications and Future Agendas. IEEE Transactions on Intelligent Transportation Systems 25:6 ► pp. 4918 ff. DOI logo
Wang, Dingyu, Shaocheng Jia, Xin Pei, Chunyang Han, Danya Yao & Dezhi Liu
2024. DERNet: Driver Emotion Recognition Using Onboard Camera. IEEE Intelligent Transportation Systems Magazine 16:2 ► pp. 117 ff. DOI logo
Zhou, Xin, Xing Chen, Liu Tang, Yi Wang, Jingyue Zheng & Wei Zhang
2024. Event-related driver stress detection with smartphones in an urban environment: a naturalistic driving study. Ergonomics 67:10 ► pp. 1371 ff. DOI logo
Bosch, Esther, Raquel Le Houcq Corbí, Klas Ihme, Stefan Hörmann, Meike Jipp & David Käthner
2023.  Frustration Recognition Using Spatio Temporal Data: A Novel Dataset and GCN Model to Recognize In-Vehicle Frustration. IEEE Transactions on Affective Computing 14:4 ► pp. 2864 ff. DOI logo
Bosch, Esther, David Käthner, Meike Jipp, Uwe Drewitz & Klas Ihme
2023. Fifty shades of frustration: Intra- and interindividual variances in expressing frustration. Transportation Research Part F: Traffic Psychology and Behaviour 94 ► pp. 436 ff. DOI logo
Rao, Shruti, Surjya Ghosh, Gerard Pons Rodriguez, Thomas Röggla, Pablo Cesar & Abdallah El Ali
2023. From Video to Hybrid Simulator: Exploring Affective Responses toward Non-Verbal Pedestrian Crossing Actions Using Camera and Physiological Sensors. International Journal of Human–Computer Interaction 39:16 ► pp. 3213 ff. DOI logo
Zhao, Sicheng, Xiaopeng Hong, Jufeng Yang, Yanyan Zhao & Guiguang Ding
2023. Toward Label-Efficient Emotion and Sentiment Analysis. Proceedings of the IEEE 111:10 ► pp. 1159 ff. DOI logo
Bosch, Esther, Marie Klosterkamp, Angelo Guevara, David Kaethner, Alexandra Bendixen & Klas Ihme
2022. 2022 13th IEEE International Conference on Cognitive Infocommunications (CogInfoCom), ► pp. 000011 ff. DOI logo
Nadri, Chihab, Ignacio Alvarez, Esther Bosch, Michael Oehl, Michael Braun, Jennifer Healey, Christophe Jallais, Wendy Ju, Jingyi Li & Myounghoon Jeon
2022. CHI Conference on Human Factors in Computing Systems Extended Abstracts, ► pp. 1 ff. DOI logo
Oh, Geesung, Euiseok Jeong, Rak Chul Kim, Ji Hyun Yang, Sungwook Hwang, Sangho Lee & Sejoon Lim
2022. Multimodal Data Collection System for Driver Emotion Recognition Based on Self-Reporting in Real-World Driving. Sensors 22:12 ► pp. 4402 ff. DOI logo
Siddiqi, Muhammad Hameed, Khalil Khan, Rehan Ullah Khan & Amjad Alsirhani
2022. Face Image Analysis Using Machine Learning: A Survey on Recent Trends and Applications. Electronics 11:8 ► pp. 1210 ff. DOI logo
Sukhavasi, Susrutha Babu, Suparshya Babu Sukhavasi, Khaled Elleithy, Ahmed El-Sayed & Abdelrahman Elleithy
2022. Deep Neural Network Approach for Pose, Illumination, and Occlusion Invariant Driver Emotion Detection. International Journal of Environmental Research and Public Health 19:4 ► pp. 2352 ff. DOI logo
Zhou, Xin, Liang Ma & Wei Zhang
2022. Event-related driver stress detection with smartphones among young novice drivers. Ergonomics 65:8 ► pp. 1154 ff. DOI logo
Bustos, Cristina, Neska Elhaouij, Albert Solé-Ribalta, Javier Borge-Holthoefer, Agata Lapedriza & Rosalind Picard
2021. 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII), ► pp. 1 ff. DOI logo
Jipp, Meike & Jochen Steil
2021. Steuern wir oder werden wir gesteuert? Chancen und Risiken von Mensch-Technik-Interaktion. In Zusammenwirken von natürlicher und künstlicher Intelligenz, ► pp. 17 ff. DOI logo
Krüger, Sandra, Esther Bosch, Klas Ihme & Michael Oehl
2021. In-Vehicle Frustration Mitigation via Voice-User Interfaces – A Simulator Study. In HCI International 2021 - Posters [Communications in Computer and Information Science, 1421], ► pp. 241 ff. DOI logo
Lemmer, Karsten, Meike Jipp, Heiner Bubb, Hans-Jörg Vögel, Matthias Jung, Georg Laukart & Thomas Vorberg
2021. Mensch-Technik-Kooperation und Fahrzeuginnenraum. In Vieweg Handbuch Kraftfahrzeugtechnik, ► pp. 1161 ff. DOI logo
Franz, Oliver, Uwe Drewitz & Klas Ihme
2020. Facing Driver Frustration: Towards Real-Time In-Vehicle Frustration Estimation Based on Video Streams of the Face. In HCI International 2020 - Posters [Communications in Computer and Information Science, 1226], ► pp. 349 ff. DOI logo
Nadri, Chihab, Jingyi Li, Esther Bosch, Michael Oehl, Ignacio Alvarez, Michael Braun & Myounghoon Jeon
2020. 12th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ► pp. 106 ff. DOI logo
Ortoncelli, Andre Roberto, Luciano Silva, Olga Regina Perreira Bellon, Tiago Mota de Oliveira & Juliana Daga
2020. 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), ► pp. 587 ff. DOI logo
Pape, Anna-Antonia, Sonja Cornelsen, Victor Faeßler, Klas Ihme, Michael Oehl, Uwe Drewitz, Franziska Hartwich, Frank Schrödel, Andreas Lüdtke & Martin Schramm
2020. Empathic assistants – Methods and use cases in automated and non-automated driving. In 20. Internationales Stuttgarter Symposium [Proceedings, ], ► pp. 435 ff. DOI logo
Weidemann, Alexandra & Nele Russwinkel
2019. Proceedings of Mensch und Computer 2019, ► pp. 819 ff. DOI logo
Weidemann, Alexandra & Nele Rußwinkel
2021. The Role of Frustration in Human–Robot Interaction – What Is Needed for a Successful Collaboration?. Frontiers in Psychology 12 DOI logo
Zepf, Sebastian, Tobias Stracke, Alexander Schmitt, Florian van de Camp & Juergen Beyerer
2019. 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), ► pp. 202 ff. DOI logo
Bosch, Esther, Michael Oehl, Myounghoon Jeon, Ignacio Alvarez, Jennifer Healey, Wendy Ju & Christophe Jallais
2018. Adjunct Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, ► pp. 44 ff. DOI logo

This list is based on CrossRef data as of 17 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue