Cover not available

In:Less Frequently Used Research Methodologies in Applied Linguistics
Edited by A. Mehdi Riazi
[Research Methods in Applied Linguistics 6] 2024
► pp. 49–67

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 4
Multimodality

A systemic-functional semiotic perspective

 | National Institute of Education
 | National Institute of Education
 | National Institute of Education
Published online: 5 January 2024
DOI logo with link
https://doi.org/10.1075/rmal.6.04che
Abstract
Multimodality examines how language and other resources (e.g., images and gestures) are integrated to make meaning for communication. This chapter aims to introduce two approaches to multimodality from a systemic-functional semiotic perspective, namely social semiotics and systemic functional multimodal discourse analysis (SF-MDA), and discuss their applications in applied linguistics. Specifically, this chapter begins with an overview of the theoretical underpinnings of the two approaches, focusing on their shared functionalist origins and the origin’s theoretical and methodological implications. Next, the chapter discusses the affordance of social semiotics and SF-MDA and highlights data collection procedures and analytical processes in applied linguistic studies informed by the two approaches. Finally, critiques of the two approaches are addressed with multimodal research facilitated by eye-tracking technology.
Article outline
  • 1.Introduction
  • 2.Theoretical foundations
    • 2.1Multimodality as a functionalist social semiotic inquiry
    • 2.2Social semiotics and systemic functional multimodal discourse analysis (SF-MDA)
  • 3.Methodological orientation
    • 3.1Principles and affordances
    • 3.2Types of research questions addressed by social semiotics and SF-MDA
    • 3.3Procedures of data collection and analysis
      • 3.3.1Data collection
      • 3.3.2Data analysis
    • 3.4Research ethics
  • 4.Systemic-functional semiotics: Critiques and responses
  • 5.Conclusions
  • References
References (70)
References
Amundrud, T. (2018). Applying multimodal research to the tertiary foreign language classroom: Looking at gaze. In H. D. S. Joyce & S. Feez (Eds), Multimodality across classrooms: Learning about and through different modalities (pp. 160–177). Routledge. Google Scholar logo with link to Google Scholar
Bateman, J. (2014). Text and image: A critical introduction to the visual/verbal divide. Routledge. Google Scholar logo with link to Google Scholar
Bateman, J. A. (2018). Towards critical multimodal discourse analysis: A response to Ledin and Machin. Critical Discourse Studies, 15(5), 1–9. Google Scholar logo with link to Google Scholar
Bateman, J. A., Beckmann, A., & Varela, R. I. (2018). From empirical studies to visual narrative organization: Exploring page composition. In A. Dunst, J. Laubrock, & J. Wildfeuer (Eds.), Empirical comics research: Digital, multimodal, and cognitive methods (pp.127–153). Routledge. Google Scholar logo with link to Google Scholar
Bateman, J., Wildfeuer, J., & Hiippala, T. (2017). Multimodality: Foundations, research and analysis. Walter de Gruyter. Google Scholar logo with link to Google Scholar
Bezemer, J., & Kress, G. (2008). Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication, 25(2), 166–195. Google Scholar logo with link to Google Scholar
(2016). Multimodality, learning and communication: A social semiotic frame. Routledge.Google Scholar logo with link to Google Scholar
Bezemer, J., & Mavers, D. (2011). Multimodal transcription as academic practice: A social semiotic perspective. International Journal of Social Research Methodology, 13(3), 191–206. Google Scholar logo with link to Google Scholar
Boeriis, M., & Holsanova, J. (2012). Tracking visual segmentation: Connecting semiotic and cognitive perspectives. Visual Communication, 11(3), 259–281. Google Scholar logo with link to Google Scholar
Bowcher, W. L., & Zhang, Z. (2020). Pointing at words: Gestures, language and pedagogy in elementary literacy classrooms in China. Linguistics and Education, 55, 100779. Google Scholar logo with link to Google Scholar
Chapelle, C. A. (2016). Teaching culture in introductory foreign language textbooks. Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Chen, Y. (2022). Salient visual foci on human faces in viewers’ engagement with advertisements: Eye-tracking evidence and theoretical implications. Multimodality and Society, 2(1), 3–22. Google Scholar logo with link to Google Scholar
Christie, F. (2002). Classroom discourse analysis: A functional perspective. Continuum.Google Scholar logo with link to Google Scholar
Conklin, K., Pellicer-Sánchez, A., & Carrol, G. (2018). Eye-tracking: A guide for applied linguistics research. Cambridge University Press. Google Scholar logo with link to Google Scholar
Cowan, K. (2014). Multimodal transcription of video: examining interaction in early years classrooms. Classroom Discourse, 5(1), 6–21. Google Scholar logo with link to Google Scholar
Darvin, R. (2020). Creativity and criticality: Reimagining narratives through translanguaging and transmediation. Applied Linguistics Review, 11(4), 581–606. Google Scholar logo with link to Google Scholar
Dik, S. C., & Hengeveld, K. (Eds.). (1997). The theory of functional grammar, Part 1: The structure of the clause. Walter de Gruyter.Google Scholar logo with link to Google Scholar
Feng, W. D. (2019). Infusing moral education into English language teaching: An ontogenetic analysis of social values in EFL textbooks in Hong Kong. Discourse: Studies in the Cultural Politics of Education, 40(4), 458–473. Google Scholar logo with link to Google Scholar
Flewitt, R. (2006). Using video to investigate preschool classroom interaction: Education research assumptions and methodological practices. Visual Communication, 5, 25–50. Google Scholar logo with link to Google Scholar
Godfroid, A. (2019). Eye tracking in second language acquisition and bilingualism: A research synthesis and methodological guide (1st ed.). Routledge. Google Scholar logo with link to Google Scholar
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. Hodder Education.Google Scholar logo with link to Google Scholar
Halliday, M. A. K., & Matthiessen, C. (2004). An introduction to functional grammar (3rd ed.). Edward Arnold.Google Scholar logo with link to Google Scholar
Holsanova, J. (2014). Reception of multimodality: Applying eye tracking methodology in multimodal research. In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp. 285–296). Routledge.Google Scholar logo with link to Google Scholar
Holsanova, J., Rahm, H., & Holmqvist, K. (2006). Entry points and reading paths on newspaper spreads: Comparing a semiotic analysis with eye-tracking measurements. Visual Communication, 55(1), 65–93. Google Scholar logo with link to Google Scholar
Hood, S. E. (2011). Body language in face-to-face teaching: A focus on textual and interpersonal meaning. In E. A. Thompson, M. Stenglin, & S. Dreyfus (Eds), Semiotic margins: Meaning in multimodalities (pp. 31–52). Continuum.Google Scholar logo with link to Google Scholar
Jewitt, C. (2009). Different approaches to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp. 28–39). Routledge.Google Scholar logo with link to Google Scholar
(2014a). An introduction to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp.15–30). Routledge.Google Scholar logo with link to Google Scholar
(2014b). Different approaches to multimodality. In C. Jewitt (Ed.). The Routledge handbook of multimodal analysis (pp. 31–43). Routledge.Google Scholar logo with link to Google Scholar
Jewitt, C., Bezemer, J., & O’Halloran, K. (2016). Introducing multimodality. Routledge. Google Scholar logo with link to Google Scholar
Jiang, L., Yang, M., & Shu, Y. (2020). Chinese ethnic minority students’ investment in English learning empowered by digital multimodal composing. TESOL Quarterly, 54(4), 954–979. Google Scholar logo with link to Google Scholar
Joo, S. J., Chik, A., & Djonov, E. (2020). The construal of English as a global language in Korean EFL textbooks for primary school children. Asian Englishes, 22(1), 68–84. Google Scholar logo with link to Google Scholar
Knoblauch, H., Schnettler, B., Raab, J., & Soeffner, H.-G. (Eds.). (2009). Video analysis: Methodology and methods; qualitative audiovisual data analysis in sociology (2nd ed.). Peter Lang.Google Scholar logo with link to Google Scholar
Korhonen, V. (2010). Dialogic literacy: A sociocultural literacy learning approach. In A. Lloyd & T. Sanna (Eds.), Practising information literacy (pp. 211–226). Chandos. Google Scholar logo with link to Google Scholar
Kress, G. (2000). Design and transformation: New theories of meaning. In B. Cope & M. Kalantzis (Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 149–158). Routledge.Google Scholar logo with link to Google Scholar
(2010). Multimodality: A social semiotic approach to contemporary communication. Routledge.Google Scholar logo with link to Google Scholar
(2014). What is mode? In C. Jewitt (Ed.), The Routledge handbook of multimodal analysis (pp.60–75). Routledge.Google Scholar logo with link to Google Scholar
Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K., & Reid, E. (2005). English in urban classrooms: A multimodal perspective on teaching and learning. Falmer.Google Scholar logo with link to Google Scholar
Kress, G., & Selander, S. (2012). Multimodal design, learning and cultures of recognition. The Internet and Higher Education, 15(4), 265–268. Google Scholar logo with link to Google Scholar
Kress, G., & van Leeuwen, T. (2001). Multimodal discourse: The modes and media of contemporary communication. Edward Arnold.Google Scholar logo with link to Google Scholar
(2006). Reading images: The grammar of visual design (2nd ed.). Routledge. Google Scholar logo with link to Google Scholar
Ledin, P., & Machin, D. (2018). Doing critical discourse studies with multimodality: From metafunctions to materiality. Critical Discourse Studies, 15(1), 1–17. Google Scholar logo with link to Google Scholar
Lemke, J. (1998). Multiplying meaning: Visual and verbal semiotics in scientific text. In J. R. Martin & R. Vell (Eds.), Reading science: Critical and function (pp. 87–113). Routledge.Google Scholar logo with link to Google Scholar
Lim, F. V. (2021a). Designing learning with embodied teaching: Perspectives from multimodality. Routledge.Google Scholar logo with link to Google Scholar
(2021b). Investigating intersemiosis: A systemic functional multimodal discourse analysis of the relationship between language and gesture in classroom discourse. Visual Communication, 20(1), 34–58. Google Scholar logo with link to Google Scholar
Lim, F. V., O’Halloran, K. L., & Podlasov, A. (2012). Spatial pedagogy: Mapping meanings in the use of classroom space. Cambridge Journal of Education, 42(2), 235–251. Google Scholar logo with link to Google Scholar
Miller, S. M., & McVee, M. B. (Eds.). (2013). Multimodal composing in classrooms: Learning and teaching for the digital world. Routledge. Google Scholar logo with link to Google Scholar
Mills, K. A. (2015). Literacy theories for the digital age: Social, critical, multimodal, spatial, material and sensory lenses. Multilingual Matters. Google Scholar logo with link to Google Scholar
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60–92. Google Scholar logo with link to Google Scholar
Norris, S. (2004). Analyzing multimodal interaction: A methodological framework. Routledge. Google Scholar logo with link to Google Scholar
O’Halloran, K. L. (2005). Mathematical discourse: Language, symbolism and visual images. Continuum.Google Scholar logo with link to Google Scholar
(2008). Systemic functional-multimodal discourse analysis (SF-MDA): Constructing ideational meaning using language and visual imagery. Visual Communication, 7(4), 443–475. Google Scholar logo with link to Google Scholar
(2011). Multimodal discourse analysis. In K. Hyland & B. Paltridge (Eds.), Companion to discourse (pp. 120–137). Continuum.Google Scholar logo with link to Google Scholar
(Ed.). (2004). Multimodal discourse analysis: Systemic-functional perspectives. Continuum.Google Scholar logo with link to Google Scholar
O’Halloran, K. L., & Lim, F. V. (2014). Systemic functional multimodal discourse analysis. In S. Norris & C. Maier (Eds.), Texts, images and interactions: A Reader in multimodality (pp. 135–154). De Gruyter. Google Scholar logo with link to Google Scholar
O’Halloran, K. L., Podlasov, A., Chua, A., & Marissa, K. L. E. (2012). Interactive software for multimodal analysis. Visual Communication, 11(3), 363–381. Google Scholar logo with link to Google Scholar
O’Toole, M. (2011). The language of displayed art (2nd ed.). Routledge.Google Scholar logo with link to Google Scholar
Pirini, J., Matelau-Doherty, T., & Norris, S. (2018). Multimodal analysis. In A. Phakiti, P. De Costa, L. Plonsky, & S. Starfield, (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 639–658). Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Ranker, J. (2017). The role of signifier differences, associations, and combinations in creative digital video composing: Making meaning with gestures, objects, actions, and speech. Pedagogies: An International Journal, 12(2), 196–218. Google Scholar logo with link to Google Scholar
Stivers, T., & Sidnell, J. (2005). Introduction: Multimodal interaction. Semotica, 156, 1–20. Google Scholar logo with link to Google Scholar
Teo, P., & Zhu, J. (2018). A multimodal analysis of affect and attitude education in China’s English textbooks. Multimodal Communication, 7(1), 20170014. Google Scholar logo with link to Google Scholar
Thomas, M. A. (2020). Formalism and functionalism in linguistics: The engineer and the collector. Routledge.Google Scholar logo with link to Google Scholar
Van Leeuwen, T. (2005). Introducing social semiotics. Routledge.Google Scholar logo with link to Google Scholar
Weninger, C. (2021). Multimodality in critical language textbook analysis. Language, Culture & Curriculum, 34(2), 133–146. Google Scholar logo with link to Google Scholar
Weninger, C. & Kiss, T. (2013). Culture in English as a Foreign Language (EFL) textbooks: A semiotic approach. TESOL Quarterly, 47(4), 694–716. Google Scholar logo with link to Google Scholar
Wigham, C., & Satar, M. (2021). Multimodal (inter)action analysis of task instructions in language teaching via videoconferencing: A case study. ReCALL, 33(3), 195–213. Google Scholar logo with link to Google Scholar
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006, May 24–26). ELAN: A professional framework for multimodality research [Conference presentation]. The International Conference on Language Resources and Evaluation, Genoa, Italy. [URL]
Xiong, T. & Peng, Y. (2021). Representing culture in Chinese as a second language textbooks: A critical social semiotic approach. Language, Culture & Curriculum, 34(2), 163–182. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue