The multimodal approach in audiovisual translation

This paper will explore the multimodal approach to audiovisual translation (AVT). It must first be stressed, however, that most research on multimodality has not as yet focused on questions of translation. The Routledge Handbook of Multimodal Analysis (Jewitt 2009Jewitt, Carey, ed. 2009. The Routledge Handbook of Multimodal Analysis. Abingdon: Routledge.), which contains articles by most of the leading figures in the field, while representing a major step forward in multimodal studies, does not tackle translation head on. The word ‘translation’ does not even appear in the index. Over a relatively short time span, most of the major contributions to the field have been more purely linguistically based and intent on providing keys to the understanding of the interplay of semiotic resources such as words, images, gesture, music, light, etc. (see O’Toole 1994O’Toole, Michael. 1994. The Language of Displayed Art. London: Leicester University Press.; Kress and van Leeuwen 1996Kress, Gunther, and Theo van Leeuwen. 1996. Reading Images: the Grammar of Visual Design. London: Routledge.; Martinec 2000Martinec, Radan. 2000. “Rhythm in Multimodal Texts.” LEONARDO 33 (4): 289–297.; Unsworth 2001Unsworth, Len. 2001. Teaching Multiliteracies across the Curriculum: Changing Contexts of Text and Image in Classroom Practice. Buckingham: Open University Press.; Baldry and Thibault 2006Baldry, Anthony, and Paul Thibault. 2006. Multimodal Transcription and Text Analysis. London: Equinox., etc.). The work of these scholars, however, has provided an impetus to developing ideas on how to exploit multimodal analyses in the area of AVT. Thibault’s work, for example, on the ‘multimodal transcription’ provided this author with the basis for investigating how the integration of semiotic modalities in a film text could assist the subtitler in making those all-important decisions on what to retain and what to discard when faced with time constraints. Other scholars have studied the co-articulation of words and image in their discussion of how different modalities realize social functions and make meaning (O’Halloran 2008O’Halloran, Kay. 2008. Mathematical Discourse: Language, Symbolism and Visual Images. London: Continuum.; Bednarek 2010Bednarek, Monica. 2010. The Language of Fictional Television: Drama and Identity. London: Continuum.), emphasising the importance of supplementing purely linguistic analyses with studies of all the other semiotic resources that make up a multimodal text. Findings will inevitably be reported verbally but the analyses need to explore the concept of integration and how other resources can interact with language and, crucially, how translators can be made sensitive to the entire semiotic impact of a multimodal text.