Cover not available

Article published In: Translation and Interpreting at the Interface of Cognition and Emotion
Edited by Ana María Rojo López and Purificación Meseguer Cutillas
[Translation, Cognition & Behavior 7:1] 2024
► pp. 48–89

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Multimodal exploration of the thank God expressive construction and its implications for translation

ORCiD logo with link | University of Murcia
Published online: 10 October 2024
DOI logo with link
https://doi.org/10.1075/tcb.00095.mar
Abstract
Multimodal research in communication and translation studies is increasingly recognized, yet it remains incompletely explored. Leveraging computational linguistics with both Praat for acoustic analysis and the OpenPose and Rapid Annotator tools for visual analysis, this study delves into the intricate dynamics of the expressive construction thank God, providing a comprehensive examination of both visual and acoustic dimensions. Our objective is to uncover nuanced patterns of multimodal communication embedded within this expression and their implications for Translation and Interpreting. Through an analysis of linguistic features and co-speech gestures present in thank God, we aim to deepen our comprehension of how meaning crisscrosses modalities. Our findings underscore the necessity of a multimodal approach in language studies, emphasizing the requisite to preserve emotional and contextual nuances. The analysis unveils the phonological relevance of the duration of the construction’s second vowel, a key factor for translation. Additionally, data reveals a correlation between the emotion of relief and gestures executed with both hands closer to the chest. Overall, these findings contribute to advancing both multimodal communication research and translation studies, shedding light on the role of multimodal analysis in understanding language and translation dynamics, particularly in the context of constructions like thank God.
Article outline
  • 1.Introduction
  • 2.The empirical study
    • 2.1Aims and hypotheses
    • 2.2Instruments
    • 2.3Materials and methods
    • 2.4Procedure
    • 2.5Selection of dependent and independent variables
    • 2.6Data processing and statistical analysis
    • 2.7Results
      • 2.7.1Results for the number of samples, frequency, and ratio
      • 2.7.2Results for the normality of the data: Shapiro-Wilk test
      • 2.7.3Results for the homoscedasticity of the data: Bartlett test
      • 2.7.4Results for the parametric technique: T-Student test
      • 2.7.5Results for the linear models
        • X-axis
        • Y-axis
        • X*Y axes
    • 2.8Discussion and conclusions
  • Notes
  • References
References (78)
References
Abelin, Åsa. 2011. “Perception of Emotional Interjections.” International Journal of Cognitive Linguistics 2 (2): 127–143.Google Scholar logo with link to Google Scholar
Abelin, Åsa, and Jens Allwood. 2002. “Cross-Linguistic Interpretation of Emotional Prosody.” Gothenburg Papers in Theoretical Linguistics 871. Accessed February 9, 2024.Google Scholar logo with link to Google Scholar
Ameka, Felix. 1992. “Interjections: The Universal Yet Neglected Part of Speech.” Journal of Pragmatics 18 (2–3): 101–118. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Bacharach, Julian. 2022. “Relief, Time-Bias, and the Metaphysics of Tense.” Synthese 200 (3): 1–22. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Baker, Mona. 1992. In Other Words. A Coursebook on Translation. London and New York: Routledge.Google Scholar logo with link to Google Scholar
Barsalou, Lawrence W. 1999. “Perceptual Symbol Systems.” Behavioral and Brain Sciences 221: 577–660. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Bartoń, Kamil. 2009. MuMIn: Multi-Model Inference (R package version 0.12.2). Accessed February 9, 2023. [URL]
Beckman, Mary E. 1986. Stress and Non-Stress Accent. Germany: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Bloomfield, Leonard. 1984. Language. 2nd ed. Chicago: The University of Chicago Press.Google Scholar logo with link to Google Scholar
Boersma, Paul, and Vincent J. van Heuven. 2001. “Speak and UnSpeak with PRAAT.” Glot International 5 (9–10): 341–347. Accessed February 9, 2024.Google Scholar logo with link to Google Scholar
Bolinger, Dwight L. 1958. “A Theory of Pitch Accent in English.” WORD 14 (2–3): 109–149. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Cao, Zhe, Simon, Tomas, Wei, Shih-En and Yaser Sheikh. 2017. “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Casanova, Fernando. 2023. Las Construcciones Expresivas desde una Perspectiva Cognitiva, Multimodal e Interlingüística: La Emoción de Alivio [The Expressive Constructions from a Cognitive, Multimodal and Crosslinguistic Perspective: The Emotion of Relief]. Doctoral Thesis, University of Murcia.
Chaume, Frederic. 2012. Audiovisual Translation: Dubbing. Manchester: St. Jerome.Google Scholar logo with link to Google Scholar
Chrabaszcz, Anna, Matthew Winn, Candise Y. Lin, and William J. Idsardi. 2014. “Acoustic Cues to Perception of Word Stress by English, Mandarin, and Russian Speakers.” Journal of Speech, Language, and Hearing Research 57 (4): 1468–1479. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Cienki, Alan. 2017. “Utterance Construction Grammar (UCxG) and the Variable Multimodality of Constructions.” Linguistics Vanguard 3 (s1): 20160048. Google Scholar logo with link to Google Scholar
Corpas, Gloria. 2001. “La traducción de unidades fraseológicas: técnicas y estrategias,” In La lingüística aplicada a finales del siglo XX. Ensayos y propuestas, edited by Isabel de la Cruz, 779–787. Alcalá: Universidad de Alcalá, vol. 21.Google Scholar logo with link to Google Scholar
Cuenca, Maria Josep. 2006. “Interjections and Pragmatic Errors in Dubbing.” Meta 51 (1): 20–35. Google Scholar logo with link to Google Scholar
Dingemanse, Mark. 2023. “Interjections.” In The Oxford Handbook of Word Classes, edited by Eva van Lier, 447–491. Oxford: Oxford University Press. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Ekman, Paul, and Friesen, Wallace V. 1975. Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. New Jersey, US: Prentice Hall.Google Scholar logo with link to Google Scholar
Esteve-Gibert, Núria, and Bahia Guellaï. 2018. “Prosody in the Auditory and Visual Domains: A Developmental Perspective.” Frontiers in Psychology 91: 338. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Fauconnier, Gilles. 1994. Mental Spaces: Aspects of Meaning Construction in Natural Language. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Forster, Iris, Susanne R. Borgwaldt, and Martin Neef. 2012. “Form Follows Function: Interjections and Onomatopoetica in Comics.” Writing Systems Research 4 (2): 122–139. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Fry, Dennis B. 1958. “Experiments in the Perception of Stress.” Language and Speech 1 (2): 126–152. Google Scholar logo with link to Google Scholar
Fuchs, Susanne, Egor Savin, Stephanie Solt, Cornelia Ebert, and Manfred Krifka. 2019. “Antonym Adjective Pairs and Prosodic Iconicity: Evidence from Letter Replications in an English Blogger Corpus.” Linguistics Vanguard 5 (1): 20180017. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Gasca, Luis, and Román Gubern. 2009. Diccionario de Onomatopeyas del Cómic [Dictionary of Comic Book Onomatopoeias] 2nd ed. Madrid: Cátedra.Google Scholar logo with link to Google Scholar
Gibbs, Raymond W. 1996. “What’s Cognitive about Cognitive Linguistics?” In Cognitive Linguistics in the Redwoods: The Expansion of a New Paradigm in Linguistics, edited by Eugene H. Casad, 27–54. Germany: De Gruyter Mouton. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Goddard, Cliff. 2013. “Interjections and Emotion (with Special Reference to Surprise and Disgust).” Emotion Review 6 (1): 53–63. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Goldberg, Adele. 1995. Constructions: A Construction Grammar Approach to Argument Structure. Chicago: University of Chicago Press.Google Scholar logo with link to Google Scholar
. 2006. Constructions at Work. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Gottlieb, Henrik. 1997. Subtitles, Translation & Idioms. University of Copenhagen: Center for Translation Studies and Lexicography.Google Scholar logo with link to Google Scholar
Guerrini, Janek. 2020. “Vowel Quality and Iconic Lengthening.” Proceedings of Sinn und Bedeutung 24 (1): 242–255. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Hill, Deborah. 1992. “Imprecatory Interjectional Expressions: Examples from Australian English.” Journal of Pragmatics 18 (2–3): 209–223. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Hoey, Elliot M. 2014. “Sighing in Interaction: Somatic, Semiotic, and Social.” Research on Language and Social Interaction 471: 175–200. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
2020. “Waiting to Inhale: On Sniffing in Conversation.” Research on Language and Social Interaction 53 (1): 118–139. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Hostetter, Autumn B., and Martha W. Alibali. 2007. “Raise your Hand if You’re Spatial: Relations between Verbal and Spatial Skills and Gesture Production.” Gesture 7 (1): 73–95. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Ibarretxe-Antuñano, Iraide, and Javier Valenzuela. 2021. Lenguaje y Cognición [Language and Cognition]. Madrid: Síntesis.Google Scholar logo with link to Google Scholar
Johnson, Mark. 1987. The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. Chicago: University of Chicago Press. Google Scholar logo with link to Google Scholar
Kendrick, Kobin H., Judith Holler, and Stephen C. Levinson. 2023. “Turn-Taking in Human Face-to-Face Interaction is Multimodal: Gaze Direction and Manual Gestures Aid the Coordination of Turn Transitions.” Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 378 (1875): 20210473. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Kress, Gunther, and Theo van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar logo with link to Google Scholar
Kristiansen, Gitten. 2006. “Towards a Usage-Based Cognitive Phonology.” International Journal of English Studies 6 (2): 107–140. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Kruger, Jan, Doherty, Stephen, Fox, Wendy, and Peter De Lissa. 2018. “Multimodal Measurement of Cognitive Load during Subtitle Processing: Same-Language Subtitles for Foreign-Language Viewers.” In Innovation and expansion in translation process research, edited by Isabel Lacruz and Riitta Jääskeläinen, 267–294. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Langacker, Ronald W. 1999. Grammar and Conceptualization. Germany: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
2008. Cognitive Grammar: A Basic Introduction. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Levinson, Stephen C. 2023. “Gesture, Spatial Cognition and the Evolution of Language.” Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences 378 (1875): 20210481. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
López García, Ángel. 2000. “Teoría Gramatical [Grammatical Theory].” In Introducción a la Lingüística Española, edited by Manuel Alvar, 7–22. Barcelona: Ariel.Google Scholar logo with link to Google Scholar
Masiola, Rosanna. 2019. Interjections, Translation, and Translanguaging. Cross-Cultural and Multimodal Perspectives. Lanham, Boulder, New York, London: Lexington Rowman Littlefield.Google Scholar logo with link to Google Scholar
McNeill, David, and Laura L. Pedelty. 1995. “Right Brain and Gesture.” In Language, Gesture, and Space, edited by Karen Emmorey, and Judy Snitzer Reilly, 63–85. Mahwah, US: Lawrence Erlbaum Associates Inc. Accessed February 9, 2024.Google Scholar logo with link to Google Scholar
Neef, Martin. 2005. “The Relation of Vowel Letters to Phonological Syllables in English and German.” In From Letter to Sound: New Perspectives on Writing Systems, edited by Martin Neef, and Beatrice Primus, 205–234. Amsterdam: John Benjamins. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Norris, Sigrid. 2004. Analyzing Multimodal Interaction. London: Routledge. Google Scholar logo with link to Google Scholar
Nygaard, Lynne. C., and Jennifer S. Queen. 2008. “Communicating Emotion: Linking Affective Prosody and Word Meaning.” Journal of Experimental Psychology. Human Perception and Performance 34 (4): 1017–1030. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Pagán Cánovas, Cristóbal, Javier Valenzuela, Daniel Alcaraz-Carrión, Inés Olza, and Michael Ramscar. 2020. “Quantifying the Speech-Gesture Relation with Massive Multimodal Datasets: Informativity in Time Expressions.” PloS One, 15 (6): e0233892. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Payrató, Lluís. (2006). “Discurso Oral y Multimodalidad: Aspectos Introductorios [Oral Discourse and Multimodality: Introductory Aspects].” Oralia: Análisis del Discurso Oral 91: 259–276. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Pekrun, Reinhard, Herbert W. Marsh, Andrew J. Elliot, Kristina Stockinger, Raymond P. Perry, Elisabeth Vogl, Thomas Goetz, Wijnand A. P. van Tilburg, Oliver Lüdtke, and Walter P. Vispoel. 2023. “A Three-Dimensional Taxonomy of Achievement Emotions.” Journal of Personality and Social Psychology 124 (1): 145–178. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Pöchhacker, Franz. 2016. Introducing Interpreting Studies. London: Routledge. Google Scholar logo with link to Google Scholar
Pouw, Wim, James P. Trujillo, and James A. Dixon. 2020. “The Quantification of Gesture-Speech Synchrony: A Tutorial and Validation of Multimodal Data Acquisition Using Device-Based and Video-Based Motion Tracking.” Behavior Research Methods 52 (2): 723–740. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Robinson, Peter, and Nick C. Ellis, eds. 2008. Handbook of Cognitive Linguistics and Second Language Acquisition. London: Routledge. Google Scholar logo with link to Google Scholar
Rohrer, Tim. 2007. “Embodiment and experientialism.” In The Oxford Handbook of Cognitive Linguistics, edited by Dirk Geeraerts y Hubert Cuyckens, 25–47. Oxford: Oxford University Press. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Rojo, Ana M., and Iraide Ibarretxe-Antuñano. 2013. Cognitive Linguistics and Translation: Advances in Some Theoretical Models and Applications. Germany: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Rojo, Ana M., and Marina Ramos Caro. 2016. “Can Emotion Stir Translation Skill? Defining the Impact of Positive and Negative Emotions on Translation Performance.” In Reembedding Translation Process Research, edited by Ricardo Muñoz Martín, 107–130. Amsterdam: John Benjamins. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Sánchez-Mompeán, Sofía. 2019. “The Translation of Pitch Movement in Dubbed Dialogue.” New Voices in Translation Studies 21 (1): 93–125. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Sauter, Disa A., Frank Eisner, Paul Ekman, and Sophie K. Scott. 2009. “Universal Vocal Signals of Emotion.” In Proceedings of the 31st Annual Meeting of the Cognitive Science Society, edited by Niels Taatgen, and Hedderik Van Rijn, 2251–2255. Cognitive Science Society. Accessed February 9, 2024. [URL]
Schoonjans, Steven. 2017. “Multimodal Construction Grammar issues are Construction Grammar issues.” Linguistics Vanguard 3 (s1). Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Schröder, Ulrike, Elisabetta Adami, and Jennifer Dailey-O’Cain, eds. 2023. Multimodal Communication in Intercultural Interaction. London: Routledge. Google Scholar logo with link to Google Scholar
Steen, Francis, and Mark B. Turner. 2013. “Multimodal Construction Grammar.” In Language and the Creative Mind, edited by Michael Borkent, Barbara Dancygiery, and Jennifer Hinnell, 255–274. Stanford, CA: CSLI Publications. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Steen, Francis, Anders Hougaard, Jungseock Joo, Inés Olza, Cristóbal Pagán Cánovas, Anna Pleshakova, Soumya Ray, Peter Uhrig, Javier Valenzuela, Jacek Woźny, and Mark B. Turner. 2018. “Toward an Infrastructure for Data-Driven Multimodal Communication Research.” Linguistics Vanguard 4 (1): 20170041. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Taylor, Brian A. 1976. “Towards a Sociolinguistic Analysis of ‘Swearing’ and the Language of Abuse in Australian English.” In Australia Talks: Essays on the Sociology of Australian, Immigrant and Aboriginal Languages, edited by Michael G. Clyne, 43–77. Canberra: Australian National University. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Tomasello, Michael. 2003. Constructing a Language: A Usage-Based Theory of Language Acquisition. Cambridge, US: Harvard University Press.Google Scholar logo with link to Google Scholar
Tong, Eddie M. W. 2015. “Differentiation of 13 Positive Emotions by Appraisals.” Cognition and Emotion 29 (3): 484–503. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Torres Sánchez, María A. 2000. La Interjección [The Interjection]. Cádiz: Universidad de Cádiz.Google Scholar logo with link to Google Scholar
Turchyn, Sergiy, Inés Olza, Cristóbal Pagán Cánovas, Francis Steen, Mark B. Turner, Javier Valenzuela, and Soumya Ray. 2018. “Gesture Annotation with a Visual Search Engine for Multimodal Communication Research.” Proceedings of the AAAI Conference on Artificial Intelligence 32 (1). Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Usmonovna, Abdullayeva H. 2022. “Linguistic Analyzes of Words Denoting Interjection in Different Languages.” International Journal of Social Science and Interdisciplinary Research 11 (11): 303–307. Accessed February 9, 2024. [URL]
Verhagen, Arie. 2000. “Concession Implies Causality, though in some Other Space.” In Cause – Condition – Concession – Contrast Cognitive and Discourse Perspectives, edited by Elizabeth Couper-Kuhlen, and Bernd Kortmann, 361–380. Germany: De Gruyter Mouton. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
. 2001. “Subordination and Discourse Segmentation Revisited, or: Why Matrix Clauses May Be More Dependent than Complements.” In Text Representation Linguistic and Psychological Aspects, edited by Ted J. M. Sanders, Joost Schilperoord, and Wilbert Spooren, 337–357. Amsterdam: John Benjamins. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
. 2002. “From Parts to Wholes and Back Again.” Cognitive Linguistics 131: 403–439. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Wagner, Petra, Zofia Malisz, and Stefan Kopp. 2014. “Gesture and Speech in Interaction: An Overview.” Speech Communication 571: 209–232. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Wierzbicka, Anna. 1992. “The Semantics of Interjections”. Journal of Pragmatics 181: 159–191. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Williamson, Rodney. 2007. “El Diseño de un Corpus Multimodal [The Design of a Multimodal Corpus]”. Estudios de Lingüística Aplicada 461: 207–231. Accessed February 9, 2024. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue