Cover not available

In:From Gesture in Conversation to Visible Action as Utterance: Essays in honor of Adam Kendon
Edited by Mandana Seyfeddinipur and Marianne Gullberg
[Not in series 188] 2014
► pp. 15–34

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Including facial gestures in gesture-speech ensembles

 | Department of Psychology, University of Victoria
 | Department of Psychology, University of Victoria
 | Department of Psychology, University of Victoria
Published online: 6 August 2014
DOI logo with link
https://doi.org/10.1075/z.188.02bav
Conversational facial gestures fit Kendon’s (2004) specifications of the functions of hand gestures. We illustrate how facial gestures in dialogue, like hand gestures, convey referential content as well as serving pragmatic, interpersonal and interactive functions. Hand and facial gestures often occur together, creating an integrated visual image in gesture–speech ensembles. A semantic features analysis demonstrates how speakers adjust their use of these visible versus audible expressive resources according to context. Speakers who were interacting face-to-face (compared to speakers who could not see their addressee) were significantly more likely to rely on their hand and facial gestures than on their words when describing key semantic features, and their gestures were more likely to convey information that was not in their words.
References (38)
References
Bavelas, J., and Chovil, N. 1997. “Faces in dialogue.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 334–346. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Bavelas, J., Chovil, N., Lawrie, D.A., and Wade, A. 1992. “Interactive gestures.” Discourse Processes 15: 469–489. Google Scholar logo with link to Google Scholar
Bavelas, J., Gerwing, J., and Healing, S. 2014. “Hand and facial gestures in conversational interaction.” In Handbook of Language and Social Psychology, T. Holtgraves (ed.), 111–130. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
. in press. “The effect of dialogue on demonstrations: direct quotations, facial portrayals, hand gestures, and figurative references.” (accepted for publication in Discourse Processes)
Bavelas, J., Gerwing, J., Sutton, C., and Prevost, D. 2008. “Gesturing on the telephone: Independent effects of dialogue and visibility.” Journal of Memory and Language 58: 495–520. Google Scholar logo with link to Google Scholar
Beattie, G., and Shovelton, H. 1999. “Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation.” Semiotica 123 (1–2): 1–30. Google Scholar logo with link to Google Scholar
. 2002. “An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.” British Journal of Psychology 93: 179–192. Google Scholar logo with link to Google Scholar
Birdwhistell, R.L.1970. “Masculinity and femininity as display.” Kinesics and Context: Essays on Body Motion. Philadelphia, PA: University of Pennsylvania.Google Scholar logo with link to Google Scholar
Brugman, H., and Russel, A. 2004. “Annotating multimedia/multimodal resources with ELAN.” In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004) , M.T. Lino, M.F. Xavier, F. Ferreira, R. Costa, and R. Silva (eds), 2065–2068. Paris: European Language Resources Association.Google Scholar logo with link to Google Scholar
Brunner, L.J. 1979. “Smiles can be back channels.” Journal of Personality and Social Psychology 37: 728–734. Google Scholar logo with link to Google Scholar
Chovil, N. 1989. Communicative Functions of Facial Displays in Conversation. Ph.D. dissertation, University of Victoria, BC.
. 1991/1992. “Discourse-oriented facial displays in conversation.” Research on Language and Social Interaction 25: 163–192. Google Scholar logo with link to Google Scholar
. 1991. Social determinants of facial displays. Journal of Nonverbal Behavior 15: 141–153. Google Scholar logo with link to Google Scholar
. 1997. “Facing others. A social communicative perspective on facial displays.” In The Psychology of Facial Expression, J.A. Russell and J.M. Fernandez-Dols (eds), 321–333. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Coates, L.J. 1991. A Collaborative Theory of Inversion: Irony in Dialogue. M.A. thesis, University of Victoria, BC.
de Ruiter, J.P., Bangerter, A., and Dings, P. 2012. “The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis.” Topics in Cognitive Science 4 (2): 232–248. Google Scholar logo with link to Google Scholar
Ekman, P. 1979. “About brows: Emotional and conversational signals.” In Human Ethology, M. von Cranach, K. Foppa, W. Lepenies and D. Ploog (eds), 169–249. Cambridge: ­Cambridge University Press.Google Scholar logo with link to Google Scholar
. 1985. Telling Lies. New York: Norton.Google Scholar logo with link to Google Scholar
. 1997. “Should we call it expression or communication?” European Journal of Social Sciences 10: 333–359.Google Scholar logo with link to Google Scholar
Ekman, P., and Friesen, W.V. 1969. “The repertoire of nonverbal behavior: Categories, origins, usage, and coding.” Semiotica I: 49–98.Google Scholar logo with link to Google Scholar
Fridlund, A.J., Ekman, P., and Oster, H. 1987. “Facial expressions of emotion: Review of literature, 1970–1983.” In Nonverbal Communication and Behavior (2nd ed.), A.W. Siegman and S. Feldstein, eds. (143–224). Hillsdale, NJ: Erlbaum.Google Scholar logo with link to Google Scholar
Holler, J., and Beattie, G. 2002. “A micro-analytic investigation of how iconic gestures and speech represent core semantic features in talk.” Semiotica 142: 31–69.Google Scholar logo with link to Google Scholar
. 2003. “How iconic gestures and speech interact in the representation of meaning: Are both aspects really integral to the process?” Semiotica 146: 81–116.Google Scholar logo with link to Google Scholar
. 2004. “The interaction of iconic gesture and speech.” In Lecture Notes in Computer Science, 5th International Gesture Workshop, Genova, Italy, 2003: Selected Revised Papers , A. Cammurri and G. Volpe (eds), 63–69. Heidelberg: Springer Verlag.Google Scholar logo with link to Google Scholar
Holler, J., and Stevens, R. 2007. “An experimental investigation into the effect of common ground on how speakers use gesture and speech to represent size information in referential communication.” Journal of Language and Social Psychology 26: 4–27. Google Scholar logo with link to Google Scholar
Holler, J., and Wilkin, K. 2009. “Communicating common ground: How mutually shared knowledge influences the representation of semantic information in speech and gesture in a narrative task.” Language and Cognitive Processes 24: 267–289. Google Scholar logo with link to Google Scholar
Holler, J., Tutton, M., and Wilkin, K. 2011. “Co-speech gestures in the process of meaning coordination.” In Proceedings of the 2nd GESPIN – Gesture in Speech and Interaction Conference , Bielefeld, 5–7, September 2011.
Kendon, A. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Kraut, R.E., and Johnston, R.E. 1979. “Social and emotional messages of smiling: An ethological approach.” Journal of Personality and Social Psychology 61: 743–745.Google Scholar logo with link to Google Scholar
Pine, K.J., Burney, D.J., and Fletcher, B. 2010. “The semantic specificity hypothesis: When gestures do not depend upon the presence of a listener.” Journal of Nonverbal Behavior 34: 169–178. Google Scholar logo with link to Google Scholar
Rimé, B. 1982. “The elimination of visible behavior from social interactions: Effects on verbal, nonverbal and interpersonal variables.” European Journal of Social Psychology 12: 113–129. Google Scholar logo with link to Google Scholar
Sherzer, J. 1973. Verbal and Nonverbal Deixis: The Pointed Lip Gesture among the San Blas Cuna. University of Texas at Austin: Institute of Latin American Studies.Google Scholar logo with link to Google Scholar
Tomasello, M., Hare, B., Lehmann, H., and Call, J. 2007. “Reliance on head versus eyes in the gaze following of great apes and human infants: the cooperative eye hypothesis.” Journal of Human Evolution 52 (3): 314–320. Google Scholar logo with link to Google Scholar
Wittenburg, P., Brugman, H., Russel, A., Klassman, A., and Sloetjes, H. 2006. “ELAN: A professional framework for multimodality research.” In Proceedings of LREC 2006, Fifth International Conference on Language Resources and Evaluation , Genoa.
Cited by (10)

Cited by ten other publications

Edman, Kristina
2024. Doing mutual understanding in child and family therapy sessions: How three interlocutors calibrate new information. Discourse Studies 26:2 ► pp. 199 ff. DOI logo
Ito, Koya, Yoko Ishii, Ryo Ishii, Shin-Ichiro Eitoku & Kazuhiro Otsuka
2024. Exploring Multimodal Nonverbal Functional Features for Predicting the Subjective Impressions of Interlocutors. IEEE Access 12 ► pp. 96769 ff. DOI logo
Imamura, Mai, Ayane Tashiro, Shiro Kumano & Kazuhiro Otsuka
2023. INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ► pp. 42 ff. DOI logo
Imamura, Mai, Ayane Tashiro, Shiro Kumano & Kazuhiro Otsuka
2025. Synergistic Functional Spectrum Analysis: A Framework for Exploring the Multifunctional Interplay Among Multimodal Nonverbal Behaviours in Conversations. IEEE Transactions on Affective Computing 16:2 ► pp. 1056 ff. DOI logo
Trujillo, James P. & Judith Holler
2023. Interactionally Embedded Gestalt Principles of Multimodal Human Communication. Perspectives on Psychological Science 18:5 ► pp. 1136 ff. DOI logo
Trujillo, James P. & Judith Holler
2025. Multimodal information density is highest in question beginnings, and early entropy is associated with fewer but longer visual signals. Discourse Processes 62:2 ► pp. 69 ff. DOI logo
Amoyal, Mary, Roxane Bertrand, Brigitte Bigi, Auriane Boudin, Christine Meunier, Berthille Pallaud, Béatrice Priego-Valverde, Stéphane Rauzy & Marion Tellier
2022. Principes et outils pour l’annotation des corpus. TIPA. Travaux interdisciplinaires sur la parole et le langage :38 DOI logo
Sharkov, F. I., V. V. Silkin & O. F. Kireeva
2022. Non-verbal signs of personality: Communicative meanings of facial expressions. RUDN Journal of Sociology 22:2 ► pp. 387 ff. DOI logo
Clift, Rebecca
2021. Embodiment in Dissent: The Eye Roll as an Interactional Practice. Research on Language and Social Interaction 54:3 ► pp. 261 ff. DOI logo
Armstrong, Meghan, Núria Esteve Gibert, Iris Hübscher, Alfonso Igualada & Pilar Prieto
2018. Developmental and cognitive aspects of children’s disbelief comprehension through intonation and facial gesture. First Language 38:6 ► pp. 596 ff. DOI logo

This list is based on CrossRef data as of 24 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue