Cover not available

Article published In: Text and Context Revisited Within a Multimodal Framework
Edited by Yves Gambier and Olli Philippe Lautenbacher
[Babel 70:1/2] 2024
► pp. 138–163

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Towards a corpus-based approach to graphic elements in creative subtitling

A case study of the YouTube channel “Apenjie with Dawang”

 | The Hong Kong Polytechnic University
Published online: 15 December 2023
DOI logo with link
https://doi.org/10.1075/babel.00382.wu
Abstract
This article explores how a corpus-based approach allows us to describe and analyze the multimodal complexity of graphic elements in creative subtitling. To this end, the article focuses on a YouTube channel, Apenjie with Dawang, featuring a dog and its owner. This channel’s subtitling strategies were experimental with multiple graphic elements (colors, positions, font sizes, and emojis). Informed by a social semiotic approach to multimodality, a corpus of 1,155 coupled pairs of Chinese-English subtitles was annotated for modal shifts and metafunctional shifts. Some major findings include: (a) emojis were much more likely to be added to the target subtitles for the animals than for the humans; (b) speaker-identifying graphic elements (color and position) were lost in the target subtitles, but emojis were systematically added to mark animals as the speakers; (c) the addition of emojis evoked complementary-interpersonal meanings, suggesting that the subtitlers might have prioritized audience engagement over textual fidelity; (d) although the target subtitles used fewer graphic elements, the semiotic meanings could be similar or complementary to those of the source subtitles. Based on these findings, the article also discusses the opportunities and challenges of a corpus-based approach to graphic elements in creative subtitling.
Résumé
Notre article se propose d’explorer la manière dont l’approche par corpus permet d’analyser la complexité multimodale des éléments graphiques du sous-titrage créatif. Pour ce faire, on a opté pour Apenjie with Dawang, un vidéoblog, représentant un chien et sa maitresse, dans lequel sont expérimentées des stratégies de sous-titrage touchant de multiples éléments graphiques (couleurs, positions, tailles des caractères et émojis). Dans la perspective de la multimodalité perçue par la sémiotique sociale, nous avons annoté 1155 sous-titres chinois-anglais pour noter les transformations modales et méta-fonctionnelles. Il en ressort que, dans les sous-titres, (a) on a ajouté plus d’émojis pour les animaux que pour les humains, (b) on a perdu couleurs et positions identifiant le locuteur même si des émojis ont été systématiquement ajoutés pour indiquer les animaux comme locuteurs, (c) en ajoutant des émojis pour marquer du sens interpersonnel-complémentaire, les sous-titreurs ont sans doute eu davantage le souci de l’engagement du public que la fidélité au texte, (d) le sens sémiotique pourrait être similaire ou complémentaire de celui des sous-titres de départ, même si les sous-titres d’arrivée ont eu recours à moins d’éléments graphiques. A partir de ces résultats, l’article discute des opportunités et défis de l’approche par corpus des éléments graphiques dans le sous-titrage créatif.
Article outline
  • 1.Introduction
  • 2.Research background
    • 2.1A social semiotic approach to multimodality
    • 2.2A corpus-based approach to subtitles
  • 3.The study
    • 3.1Corpus
    • 3.2Coding framework
    • 3.3Coding and analytical procedures
  • 4.Results
    • 4.1Within-group comparisons
    • 4.2Inter-group comparisons
  • 5.Discussion
    • 5.1Graphic elements in creative subtitling
    • 5.2Opportunities and challenges of a corpus-based approach
  • 6.Conclusion
  • Notes
  • References
References (77)
References
Primary sources
Video links for the examples in Figure 3.
Example 1: [URL] (2:44)
Example 2: [URL] (1:20)
Example 3: https://www.youtube.com/watch?v=G_rOvmYpwZ8 (0:12)
Example 4: [URL] (0:05)
Secondary sources
Agresti, Alan. 2013. Categorical Data Analysis. Hoboken: John Wiley & Sons.Google Scholar logo with link to Google Scholar
Baños, Rocío, Silvia Bruti, and Serenella Zanotti. 2013. “Corpus Linguistics and Audiovisual Translation: In Search of an Integrated Approach.” Perspectives 21 (4): 483–490. Google Scholar logo with link to Google Scholar
Bassnett, Susan, Lawrence Venuti, Jan Pedersen, and Ivana Hostová. 2022. “Translation and Creativity in the 21st Century.” World Literature Studies 14 (1): 3–17. Google Scholar logo with link to Google Scholar
Bateman, John. 2008. Multimodality and Genre: A Foundation for the Systematic Analysis of Multimodal Documents. London: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Bateman, John A. 2014. “Using Multimodal Corpora for Empirical Research.” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 238–252. London: Routledge.Google Scholar logo with link to Google Scholar
Beckman, Rachel. 2008. “An Out-of-Character Role for Subtitles.” Washington Post, 14 November. [URL]
Bernardini, Silvia, and Adriano Ferraresi. 2022. “Corpus Linguistics.” In The Routledge Handbook of Translation and Methodology, edited by Federico Zanettin and Christopher Rundle, 207–222. London: Routledge. Google Scholar logo with link to Google Scholar
Brookes, Gavin, and Tony McEnery. 2020. “Corpus Linguistics.” In The Routledge Handbook of English Language and Digital Humanities, edited by Svenja Adolphs and Dawn Knight, 378–404. London: Routledge. Google Scholar logo with link to Google Scholar
Bruti, Silvia. 2020. “Corpus Approaches and Audiovisual Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 381–395. Cham: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Bywood, Lindsay. 2019. “Testing The Retranslation Hypothesis for Audiovisual Translation: The Films of Volker Schlöndorff Subtitled into English.” Perspectives 27 (6): 815–832. Google Scholar logo with link to Google Scholar
Chen, Yuping. 2019. Translating Film Subtitles into Chinese: A Multimodal Study. Singapore: Springer. Google Scholar logo with link to Google Scholar
Cohn, Neil. 2013. “Beyond Speech Balloons and Thought Bubbles: The Integration of Text and Image.” Semiotica 1971: 35–63. Google Scholar logo with link to Google Scholar
Dahne, Annie, and Roberta Piazza. 2020. “‘Subtitles Have to Become My Ears Not My Eyes’ Pragmatic-Stylistic Choices behind Closed Captions for the Deaf and Hard of Hearing: The Example of Breaking Bad.” In Telecinematic Stylistics, edited by Christian Hoffmann and Monika Kirner-Ludwig, 285–306. London: Bloomsbury. Google Scholar logo with link to Google Scholar
Danesi, Marcel. 2017. The Semiotics of Emoji: The Rise of Visual Language in the Age of the Internet. London: Bloomsbury. Google Scholar logo with link to Google Scholar
De Linde, Zoé, and Neil Kay. 1999. “Processing Subtitles and Film Images: Hearing vs Deaf Viewers.” The Translator 5 (1): 45–60. Google Scholar logo with link to Google Scholar
Díaz Cintas, Jorge. 2018. “‘Subtitling’s a Carnival’: New Practices in Cyberspace.” The Journal of Specialised Translation 301: 127–149.Google Scholar logo with link to Google Scholar
Díaz Cintas, Jorge, and Aline Remael. 2021. Subtitling: Concepts and Practices. London: Routledge.Google Scholar logo with link to Google Scholar
Dwyer, Tessa. 2017. Speaking in Subtitles: Revaluing Screen Translation. Edinburgh: Edinburgh University Press.Google Scholar logo with link to Google Scholar
Field, Andy. 2018. Discovering Statistics Using IBM SPSS Statistics, 5th ed. Los Angeles: Sage.Google Scholar logo with link to Google Scholar
Foerster, Anna. 2010. “Towards a Creative Approach in Subtitling: A Case Study.” In New Insights into Audiovisual Translation and Media Accessibility, edited by Jorge Díaz Cintas, Anna Matamala, and Josélia Neves, 81–98. Amsterdam: Rodopi. Google Scholar logo with link to Google Scholar
Fox, Wendy. 2016. “Integrated Titles: An Improved Viewing Experience.” In Eyetracking and Applied Linguistics, edited by Silvia Hansen-Schirra and Sambor Grucza, 5–29. Berlin: Language Science Press.Google Scholar logo with link to Google Scholar
Fox, Wendy. 2018. Can Integrated Titles Improve the Viewing Experience? Investigating the Impact of Subtitling on the Reception and Enjoyment of Film Using Eye Tracking and Questionnaire Data. Berlin: Language Science Press.Google Scholar logo with link to Google Scholar
Gambier, Yves. 2003. “Introduction: Screen Transadaptation: Perception and Reception. Special Issue on Screen Translation.” The Translator 9 (2): 171–190. Google Scholar logo with link to Google Scholar
Gambier, Yves. 2013. “The Position of Audiovisual Translation Studies.” In The Routledge Handbook of Translation Studies, edited by Carmen Millán and Francesca Bartrina, 45–59. London: Routledge. Google Scholar logo with link to Google Scholar
Guillot, Marie-Noëlle. 2019. “Subtitling on the Cusp of its Futures.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 31–47. London: Routledge.Google Scholar logo with link to Google Scholar
Halliday, Michael Alexander Kirkwood. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning. London: Edward Arnold.Google Scholar logo with link to Google Scholar
Halliday, Michael Alexander Kirkwood. 2001. “Towards a Theory of Good Translation.” In Exploring Translation and Multilingual Text Production: Beyond Content, edited by Erich Steiner and Colin Yallop, 13–18. Berlin: Mouton de Gruyter. Google Scholar logo with link to Google Scholar
Halliday, Michael Alexander Kirkwood, and Christian M. I. M. Matthiessen. 2014. Halliday’s Introduction to Functional Grammar. London: Routledge. Google Scholar logo with link to Google Scholar
Hart, Christopher. 2020. “Multimodal Discourse Analysis.” In Researching Discourse: A Student Guide, edited by Christopher Hart, 143–179. London: Routledge. Google Scholar logo with link to Google Scholar
Jewitt, Carey, Jeff Bezemer, and Kay O’Halloran. 2016. Introducing Multimodality. London: Routledge. Google Scholar logo with link to Google Scholar
Kress, Gunther. 2010. Multimodality: A Social Semiotic Approach to Contemporary Communication. London: Routledge.Google Scholar logo with link to Google Scholar
Kress, Gunther. 2014. “What Is a Mode?” In The Routledge Handbook of Multimodal Analysis, edited by Carey Jewitt, 2nd ed., 60–75. London: Routledge.Google Scholar logo with link to Google Scholar
Kress, Gunther. 2020. “Transposing Meaning: Translation in a Multimodal Semiotic Landscape.” In Translation and Multimodality: Beyond Words, edited by Monica Boria, Ángeles Carreres, María Noriega-Sánchez, and Marcus Tomalin, 24–48. London: Routledge.Google Scholar logo with link to Google Scholar
Kress, Gunther, and Theo van Leeuwen. 2006. Reading Images: The Grammar of Visual Design, 2nd ed. London: Routledge. Google Scholar logo with link to Google Scholar
Kruger, Jan-Louis. 2021. “Translation, Multimodality and Cognition.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 433–444. London: Routledge.Google Scholar logo with link to Google Scholar
Levshina, Natalia. 2017. “Online Film Subtitles as A Corpus: An N-Gram Approach.” Corpora 12 (3): 311–338. Google Scholar logo with link to Google Scholar
Logi, Lorenzo, and Michele Zappavigna. 2021. “A Social Semiotic Perspective on Emoji: How Emoji and Language Interact to Make Meaning in Digital Messages.” New Media & Society 25 (12): 3222–3246. Google Scholar logo with link to Google Scholar
Malamatidou, Sofia. 2018. Corpus Triangulation: Combining Data and Methods in Corpus-Based Translation Studies. London: Routledge.Google Scholar logo with link to Google Scholar
Matthiessen, Christian M. I. M. 2021. “Translation, Multilingual Text Production and Cognition Viewed in Terms of Systemic Functional Linguistics.” In The Routledge Handbook of Translation and Cognition, edited by Fabio Alves and Arnt Lykke Jakobsen, 517–544. London: Routledge.Google Scholar logo with link to Google Scholar
McClarty, Rebecca. 2012. “Towards A Multidisciplinary Approach in Creative Subtitling.” MonTI: Monografías de traducción e interpretación 41: 133–153. Google Scholar logo with link to Google Scholar
McClarty, Rebecca. 2014. “In Support of Creative Subtitling: Contemporary Context and Theoretical Framework.” Perspectives 22 (4): 592–606. Google Scholar logo with link to Google Scholar
McEnery, Tony, and Andrew Hardie. 2012. Corpus Linguistics: Method, Theory and Practice. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Miles, Matthew B., and A. Michael Huberman. 1994. Qualitative Data Analysis: An Expanded Sourcebook. 2 ed. Thousand Oaks, CA: Sage.Google Scholar logo with link to Google Scholar
Mubenga, Kajingulu. 2009. “Towards a Multimodal Pragmatic Analysis of Film Discourse in Audiovisual Translation.” Meta 54 (3): 466–484. Google Scholar logo with link to Google Scholar
Nornes, Abé Mark. 1999. “For an Abusive Subtitling.” Film Quarterly 52 (3): 17–34. Google Scholar logo with link to Google Scholar
Pavesi, Maria. 2019. “Corpus-Based Audiovisual Translation Studies: Ample Room for Development.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 315–333. London: Routledge.Google Scholar logo with link to Google Scholar
Pavesi, Maria. 2022. “Corpora and The Language of Films: Exploring Dialogue in English and Italian.” In The Routledge Handbook of Corpus Linguistics, 2nd ed., edited by Anne O’Keeffe and Michael J. McCarthy, 547–561. London: Routledge. Google Scholar logo with link to Google Scholar
Perego, Elisa. 2019. “Into the Language of Museum Audio Descriptions: A Corpus-Based Study.” Perspectives 27 (3): 333–349. Google Scholar logo with link to Google Scholar
Pérez-González, Luis. 2007. “Appraising Dubbed Conversation: Systemic Functional Insights into the Construal of Naturalness in Translated Film Dialogue.” The Translator 13 (1): 1–38. Google Scholar logo with link to Google Scholar
Pérez-González, Luis. 2014. Audiovisual Translation: Theories, Methods and Issues. London: Routledge. Google Scholar logo with link to Google Scholar
Pinnow, Rachel J. 2015. “A Patriot is Respectful: (Re-)Examining the Architecture of Ideology in Educational Contexts.” In International Handbook of Semiotics, edited by Peter Pericles Trifonas, 1249–1269. New York: Springer. Google Scholar logo with link to Google Scholar
Remael, Aline, and Nina Reviers. 2020. “Media Accessibility and Accessible Design.” In The Routledge Handbook of Translation and Technology, edited by Minako O’Hagan, 482–497. London: Routledge.Google Scholar logo with link to Google Scholar
Reviers, Nina. 2018. “Studying the Language of Dutch Audio Description: An Example of a Corpus-Based Analysis.” Translation and Translanguaging in Multilingual Contexts 41: 178–202. Google Scholar logo with link to Google Scholar
Romero-Fresco, Pablo. 2009. “Naturalness in the Spanish Dubbing Language: A Case of Not-so-close Friends,” Meta 54 (1), 49–72. Google Scholar logo with link to Google Scholar
Secară, Alina. 2011. “RU Ready 4 New Subtitles? Investigating the Potential of Social Translation Practices and Creative Spellings.” Linguistica Antverpiensia, New Series–Themes in Translation Studies 101: 153–171.Google Scholar logo with link to Google Scholar
Soffritti, Marcello. 2019. “Multimodal Corpora in Audiovisual Translation Studies.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 334–349. London: Routledge.Google Scholar logo with link to Google Scholar
Sun, Juhyung, Sarah Lasser, and Sun Kyong Lee. 2022. “Understanding Emojis: Cultural Influences in Interpretation and Choice of Emojis.” Journal of International and Intercultural Communication 16 (3): 246–261.Google Scholar logo with link to Google Scholar
Taylor, Christopher. 2003. “Multimodal Transcription in the Analysis, Translation and Subtitling of Italian Films.” The Translator 9 (2): 191–205. Google Scholar logo with link to Google Scholar
Taylor, Christopher. 2013. “Multimodality and Audiovisual Translation.” In Handbook of Translation Studies, vol. 41, edited by Yves Gambier and Luc van Doorslaer, 98–104. Amsterdam and Philadelphia: John Benjamins Publishing Company. Google Scholar logo with link to Google Scholar
Taylor, Christopher. 2020. “Multimodality and Intersemiotic Translation.” In The Palgrave Handbook of Audiovisual Translation and Media Accessibility, edited by Łukasz Bogucki and Mikołaj Deckert, 83–99. Cham: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Thompson, Paul. 2022. “Building A Specialised Audiovisual Corpus.” In The Routledge Handbook of Corpus Linguistics (2nd edition), edited by Anne O’Keeffe and Michael J. McCarthy, 75–88. London: Routledge. Google Scholar logo with link to Google Scholar
Tirkkonen-Condit, Sonja, and Jukka Mäkisalo. 2007. “Cohesion in Subtitles: A Corpus-Based Study.” Across Languages and Cultures 8 (2): 221–230. Google Scholar logo with link to Google Scholar
Tognini-Bonelli, Elena. 2001. Corpus Linguistics at Work. Amsterdam and Philadelphia: John Benjamins Publishing Company. Google Scholar logo with link to Google Scholar
Toury, Gideon. 2012. Descriptive Translation Studies and Beyond, rev. ed. Amsterdam and Philadelphia: John Benjamins Publishing Company. Google Scholar logo with link to Google Scholar
Tuominen, Tiina, Catalina Jiménez Hurtado, and Anne Ketola. 2018. “Why Methods Matter: Approaching Multimodality in Translation Research.” Linguistica Antverpiensia: New Series–Themes in Translation Studies 171: 1–21.Google Scholar logo with link to Google Scholar
Valentini, Cristina. 2007. “Developing AVT Corpora for a Quantitative Approach to Language Transfer in Cinematic Products.” In Selected Papers of the CETRA Research Seminar in Translation Studies 2006, edited by Francis Mus, 1–18. Leuven: CETRA.Google Scholar logo with link to Google Scholar
Wu, Zhiwei, and Zhuojia Chen. 2022. “Towards a Corpus-Driven Approach to Audiovisual Translation (AVT) Reception: A Case Study of YouTube Viewer Comments.” The Journal of Specialised Translation 381: 128–154.Google Scholar logo with link to Google Scholar
Zanettin, Federico. 2012. Translation-Driven Corpora: Corpus Resources for Descriptive and Applied Translation Studies. Manchester: St. Jerome.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue