Semiotic work: Applied Linguistics and a social semiotic account of Multimodality

Kress, Gunther

doi:10.1075/aila.28.03kre

Article published In: Theory in Applied Linguistics Research: Critical approaches to production, performance and participation
Edited by Theresa Lillis
[AILA Review 28] 2015
► pp. 49–71

Get fulltext from our e-platform

Download PDF

Semiotic work

Applied Linguistics and a social semiotic account of Multimodality

Gunther Kress | Institute of Education

Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 14 September 2015

https://doi.org/10.1075/aila.28.03kre

This article imagines a tussle between Multimodality, focused on ‘modes’, and Applied Linguistics (AL), based on ‘language’. A Social Semiotic approach to MM treats speech and writing as modes with distinct affordances, and, as all modes, treats them as ‘partial’ means of communication. The implications of partiality confound long-held assumptions of the sufficiency of ‘language’ for all communicational needs: an assumption shared by AL. Given MM’s plurality of modes and the diversity of audiences, design moves into focus, with a shift from competent performance to apt design. Principles of composition — e.g. linearity versus modularity — become crucial, raising the question at the heart of this paper: how do AL and MM deal with the shape of the contemporary semiotic landscape?

Keywords: social semiotics, multimodality, modularity, audience, writing, linearity, design, affordances

References (34)

Coulthard, M. 1992. Forensic Discourse Analysis. Advances in Spoken Discourse Analysis. London: Routledge,

Ellis, R. 2008. The Study of Second Language Acquisition. Oxford: OUP.

Fisiak, J. 1981. Contrastive Linguistics and the Language Teacher [Language Teaching Methodology Series]. Oxford: Pergamon Press.

Hartmann, R.R.K. (ed) 2003. Lexicography. Critical Concepts. London: Routledge.

König, E. & Gast, V. 2009. Understanding English-German Contrasts. Berlin: Erich Schmidt.

Kress, G. 2010. Multimodality. A Social Semiotic Approach to Contemporary Communication. London: Routledge.

Kress, G. & van Leeuwen, T. 2001. Multimodal Discourse. London: Edward Arnold.

Kress, G., Jewitt, C., Ogborn, J. & Tsatsarelis, C. 2001. Multimodal Teaching and Learning. The Rhetorics of the Science Classroom. London: Continuum.

Lado, R. 1957. Second Language Acquisition. Ann Arbor MI: University of Michigan.

Here I list key works which have informed my discussion of Multimodality in this paper.

Barthes, R. 1973. Mythologies. St. Albans: Paladin.

Bauman, Z. 2000. Liquid Modernity. Cambridge: Polity Press.

Bezemer, J. & Kress, G. 2015. The textbook in a changing multimodal landscape. In Language in Multimodal Contexts, N. Klug & H. Stöckl (eds). Berlin: De Gruyter.

Bezemer, J., Cope, A., Kress, G. & Kneebone, R. 2014. Holding the scalpel: Achieving surgical care in a learning environment. Journal of Contemporary Ethnography 43(1): 38–63.

Bezemer, J. & Kress, G. 2014. Touch: A resource for making meaning. Australian Journal of Language and Literacy 37(2): 77–85.

Bezemer, J., Kress, G., Cope, A. & Kneebone, R. 2012. Learning in the operating theatre: A social semiotic perspective. In Work-based Learning in Clinical Settings: Insights from Socio-cultural Perspectives, V. Cook, C. Daly & M. Newman (eds), 125–141. Abingdon: Radcliffe.

Bezemer, J. 2008. Displaying orientation in the classroom: Students’ multimodal responses to teacher instructions. Linguistics and Education 19(2): 166–178.

Bezemer, J. & Kress, G. 2008. Writing in multimodal texts: A social semiotic account of designs for learning. Written Communication 25(2): 166–195.

Franks, A. 1996. Drama education, the body and representation (or, the mystery of the missing bodies). Research in Drama Education: The Journal of Applied Theatre and Performance 11: 105–119.

Jewitt, C. (ed.) 2014. The Routledge Handbook of Multimodal Analysis. London: Routledge.

Kress, G. 1971. Sentence complexity in contrastive linguistics. In (ERIC) Papers in Contrastive Linguistics, G. Nickel (ed.). Cambridge: CUP.

. 1997. Before Writing: Rethinking the Paths to Literacy. London: Routledge.

. 2000. Text as the punctuation of semiosis: Pulling at some of the threads. In Intertextuality and the Media. From Genre to Everyday Life, U.H. Meinhof & J. Smith (eds), 132–153. Manchester: Manchester University Press.

Kress, G. & van Leeuwen, T. 2006. Reading Images: A Grammar of Visual Design. London: Routledge.

Kress, G., Jewitt, C., Bourne, J., Franks, A., Hardcastle, J., Jones, K. & Reid, E. 2005. English in Urban Classrooms: Multimodal Perspectives on Teaching and Learning. London: RoutledgeFalmer.

Lindstrand, F. & Insulander, E. 2014. Setting the ground for engagement: Multimodal perspectives on exhibition design. Designs for Learning 5(1-2): 30–49.

Mavers, D. 2011. Children’s Drawing and Writing: The Remarkable in the Unremarkable. London: Routledge.

New London Group. 1996. A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review 66(1): 60–92.

Newfield, D. 2014. Transformation, transduction and the transmodal moment. In The Routledge Handbook of Multimodal Analysis, C. Jewitt (ed.), 100–113. London: Routledge.

Pelletier, C. 2007. Learning through design: Subjectivity and meaning in young people’s computer game production work. PhD dissertation, University of London.

Stein, P. 2008. Multimodal Pedagogies in diverse Classrooms. Representation, Rights and Resources. New York NY: Routledge.

Unsworth, L. (ed.) 2008. Multimodal Semiotics: Functional Analysis in Contexts of Education. London: Continuum.

Wulf, C. et al. 2010. Ritual and Identity. The Staging and Performing of Rituals in the Lives of Young People. London: Tufnell.

Yandell, J. 2013. The Social Construction of Meaning. Reading Literature in Urban English Classrooms. London: Routledge.

Cited by (80)

Cited by 80 other publications

Order by:

Chen, Yixiong & Csilla Weninger

2025. The potential of eye tracking data to strengthen CDA’ explanatory power: the case of multimodal critical discourse analysis of advertising persuasion. Critical Discourse Studies 22:4 ► pp. 413 ff.

Deng, Yaling & Jinxin Jiang

2025. Emojis as mediators of digital communication: a social semiotic analysis of Taylor Swift’s Instagram posts. Popular Communication 23:3 ► pp. 179 ff.

Detwiler, Liv Halaas

2025. Interpreting teacher talk in the Spanish language classroom: Teachers’ reflections on their beliefs and practice. Pedagogies: An International Journal 20:4 ► pp. 564 ff.

Guberman, Ainat, Tal Carmi & Rachel Cohen-Brandeis

2025. Devising writing instruction methods for deaf students in a teachers’ Professional Learning Community. European Journal of Teacher Education ► pp. 1 ff.

Gómez-Victoria, Nora Lucía, Katerine Carvajal, Sebastián Collazos, Yerly Muñoz & Angie Quinchia

2025. Rethinking Communication in Interactions with a Child with Autism: Ela, Her Father, and the Jellyfish in the Pool. Scandinavian Journal of Disability Research 27:1 ► pp. 44 ff.

Lee-Pang, Lynn

2025. Meaning expression through music in teacher–child interactions. In The Arts and Meaning-Making with Children, ► pp. 92 ff.

Legel, Mascha, Stijn R. J. M. Deckers, Gloria Soto, Nicola Grove, Annalu Waller, Hans van Balkom, Ronald Spanjers, Christopher S. Norrie & Bert Steenbergen

2025. Self-Created Film as a Resource in a Multimodal Conversational Narrative. Multimodal Technologies and Interaction 9:3 ► pp. 25 ff.

Legel, Mascha, Gloria Soto, Nicola Grove, Annalu Waller, Stijn Deckers, Hans van Balkom, Ronald Spanjers, Christopher Norrie & Bert Steenbergen

2025. A tutorial: self-created film as a semiotic resource in AAC. Augmentative and Alternative Communication 41:2 ► pp. 87 ff.

Menagarishvili, Olga, Rebecca E. Burnett & Andy Frazee

2025. Configurations and Modalities: Student Preferences about Individual/Collaborative Work and In-Person/Online Work in Linked Courses. Technical Communication Quarterly ► pp. 1 ff.

Midgette, Ekaterina, Olivia G. Stewart & Ian August

2025. Understanding Multimodal Assessment Practices in Higher Education to Improve Equity. Education Sciences 15:11 ► pp. 1523 ff.

Pereira, Íris Susana Pires, Maitê Gil & Clecio Bunzen

2025. Wordless Picturebooks as Resources for the Construction of the Pedagogy of Multiliteracies. The Case of Migrants, by Issa Watanabe. Children's Literature in Education 56:1 ► pp. 69 ff.

Schur, Yaron, Ainat Guberman & Svetlana Ovsyannikov

2025. Promoting Understanding of the Concept of the Refraction of Light Through the Use of Attentive Teaching. Education Sciences 15:2 ► pp. 118 ff.

Sergeev, Dmitrii

2025. Latina American and Caribbean Single Mothers in Anglophone Children’s Picturebooks. Children's Literature in Education 56:2 ► pp. 225 ff.

Suvorov, Ruslan & Pelin Irgin

2025. Technology in Listening Assessment. In The Handbook of Second Language Listening, ► pp. 185 ff.

Jorge Eduardo Urueña López, Jamin Pelkey & L’udmila Lacková Bennett

2025. Semiosis as creation between nature, body, and art,

Xiong, Yuan & Leonardo O. Munalim

2025. Relational Processes and Social Projections in Facebook Selfie-Quotation Juxtaposition: An Exploratory Study. Human Affairs 35:1 ► pp. 83 ff.

Chen, Yixiong, Csilla Weninger & Fei Victor Lim

2024. Multimodality. In Less Frequently Used Research Methodologies in Applied Linguistics [Research Methods in Applied Linguistics, 6], ► pp. 49 ff.

Govender, Navan, Saima Salehjee & Clinton D. van der Merwe

2024. Doing critical disciplinary literacies in teacher education: A pedagogical framework. African Journal of Teacher Education and Development 3:1

Katsampoxaki-Hodgetts, Kallia

2024. Graphical abstracts’ pedagogical implications: Skills & challenges in visual remediation. English for Specific Purposes 73 ► pp. 141 ff.

Kochanowska, Anna

2024. L’émergence d’un nouveau modèle de traduction – théorie et pratique de la traduction multimodale sur l’exemple de la traduction en français de l’album Kłopot d’Iwona Chmielewska. Studia Romanica Posnaniensia 51:3 ► pp. 47 ff.

Liu, Huidan, Lihua Liu & Huadong Li

2024. Multimodal Discourse Studies in the International Academic Community (1997–2023): A Bibliometric Analysis. Sage Open 14:4

Martin, J. R.

2024. Intradisciplinarity: can one theory do it all?. Frontiers in Communication 8

Paiva, Francis Arthuso & Vicente de Lima Neto

2024. Análise sociossemiótica multimodal do grid e do layout de memes: transformações e reproduções em disputa. Revista Brasileira de Linguística Aplicada 24:4

Porto, Melina

2024. Combining linguistic and non-linguistic expression in reading in the performative university. Journal of Further and Higher Education 48:9-10 ► pp. 847 ff.

Porto, Melina

2025. Multimodal artistic recreations of the ecological university in the Global South. Pedagogy, Culture & Society 33:1 ► pp. 199 ff.

Porto, Melina

2025. Practices of walking and belonging within a multilingual Mapuche community in Argentinian Patagonia: fostering eco-consciousness in a rural primary school. International Journal of Bilingual Education and Bilingualism 28:10 ► pp. 1248 ff.

Porto, Melina

2025. How Can Language Education Contribute to Securing a Livable Planet?. TESOL Quarterly 59:2 ► pp. 585 ff.

Quintero, Edgar Nunes, Catarina Lelis & Bernardo Meza Guzman

2024. The Strange and Not So Strange About Disney’s Strange World: A Visceral and Social Semiotic Analysis. In Advances in Design and Digital Communication IV [Springer Series in Design and Innovation, 35], ► pp. 345 ff.

Rahardi, R Kunjana, Wahyudi Rahmat, Refa Lina Tiawati & Yuliana Setyaningsih

2024. Unraveling the psychological impact of spatial cybertext environments on speech intent. AILA Review

Schur, Yaron & Ainat Guberman

2024. Conceptual Change of ‘Teaching’ among Experienced Teachers after Studying Attentive Teaching. Education Sciences 14:3 ► pp. 231 ff.

Souza, Diones Bezerra de & Maria Margarete Fernandes de Sousa

2024. Multimodalidade e argumentação em um anúncio publicitário veiculado no Instagram. Diálogo das Letras 13 ► pp. e02413 ff.

Thurlow, Crispin

2024. Working beside/s words: A case study in the partiality and provinciality of language. Multimodality & Society 4:1 ► pp. 11 ff.

Wagner, Marie L.

2024. Deconstructing Identity: Empowering Student Diversity Through Critical Literacy for Enhanced Transitions. TEACHING Exceptional Children

Wyse, Dominic & Charlotte Hacking

2024. Decoding, reading and writing: the double helix theory of teaching. Literacy 58:3 ► pp. 256 ff.

Govender, Navan

2023. Critical Literacies and the Conditions of Decolonial Possibility. In Young People Shaping Democratic Politics, ► pp. 235 ff.

Govender, Navan & Jennifer Farrar

2023. Introduction to critical literacies & social media. English in Education 57:4 ► pp. 252 ff.

Guberman, Ainat

2023. Nurturing Writing of Narrative and Expository Texts at the Preschool Level. In Development of Writing Skills in Children in Diverse Cultural Contexts, ► pp. 243 ff.

Ly, Chu N. & Elena E. Forzani

2023. Let's Learn from Them: Using the Integrative Multimodal Literacy Assessment Tool to Support Instruction for Young Children. The Reading Teacher 77:3 ► pp. 407 ff.

Pascual, Daniel, Ramón Plo-Alastrué & Isabel Corona

2023. Digital Scholarly Practices in Scientific Communication: Paths and Goals in Research Dissemination. In Digital Scientific Communication, ► pp. 3 ff.

Pereira, Íris Susana Pires

2023. Focalization and ideological meanings in a digital multimodal narrative: implications for the construction of the pedagogy of multiliteracies. Multimodal Communication 12:2 ► pp. 123 ff.

Pires Pereira, Íris Susana

2022. A multiliteracies approach to online reading to learn: a case study. Pedagogies: An International Journal 17:2 ► pp. 119 ff.

Pires Pereira, Íris Susana & Ângela Campos

2023. Constructing the pedagogy of multiliteracies. The role of focalisation in the development of critical analysis of multimodal narratives. Language and Education 37:1 ► pp. 105 ff.

Tan, Lynde & Tingjia Wang

2023. Fostering Student Agency and Teacher Leadership Through Augmented Reality-Mediated Learning. In Using Innovative Literacies to Develop Leadership and Agency [Advances in Media, Entertainment, and the Arts, ], ► pp. 100 ff.

Ávila-López, Javier & Fernando D. Rubio-Alcalá

2023. Multiliteracies and multilingualism in adult migrants. European Journal of Language Policy 15:2 ► pp. 167 ff.

Bateman, John A

2022. Multimodality, where next?–Some meta-methodological considerations. Multimodality & Society 2:1 ► pp. 41 ff.

Dias, Reinildes & Daniela Gomes de Araújo Nóbrega

2022. MULTILITERACIES AND LEARNING: AN ANALYSIS OF AN ENGLISH HIGH SCHOOL TEXTBOOK FOR THE BRAZILIAN CONTEXT. Open Minds International Journal 3:1 ► pp. 80 ff.

Ding, Ai-Chu Elisha, Krista Glazewski & Faridah Pawan

2022. Language teachers and multimodal instructional reflections during video-based online learning tasks. Technology, Pedagogy and Education 31:3 ► pp. 293 ff.

Forzani, Elena E. & Chu N. Ly

2022. Beyond Multimodality to Multiplicity: Developing More Equitable and Relevant Literacy Learning Spaces for Young Children. The Reading Teacher 75:5 ► pp. 611 ff.

Kendrick, Maureen

2022. Multimodality and Multilingualism. In Multilingualism and Education, ► pp. 168 ff.

Zhu, Jianbin & Yingying Gu

2022. The Effects of Multimodal Communication Classroom on the Learning Performance of EFL Learners from the Perspective of Translanguaging. International Journal of Education and Humanities 4:3 ► pp. 223 ff.

de Souza Júnior, Jaime

2021. Transmediatisation of the Covid-19 crisis in Brazil: The emergence of (bio-/geo-)political repertoires of (re-)interpretation. Humanities and Social Sciences Communications 8:1

Li, Luyao & Jing Huang

2021. The construction of heterogeneous and fluid identities. Internet Pragmatics 4:2 ► pp. 219 ff.

Mabrook, Radwa

2021. Between Journalist Authorship and User Agency: Exploring the Concept of Objectivity in VR Journalism. Journalism Studies 22:2 ► pp. 209 ff.

Wiśniewska, Danuta

2021. Linguistic landscape, murals and language learning. In Angewandte Linguistik – Neue Herausforderungen und Konzepte, ► pp. 429 ff.

Azaoui, Brahim & Marion Tellier

2020. Comment le corps coconstruit les discours et le sens. TIPA. Travaux interdisciplinaires sur la parole et le langage :36

Govender, Navan N.

2020. Critical Transmodal Pedagogies: Student Teachers Play with Genre Conventions. Multimodal Communication 9:1

Grapin, Scott E. & Lorena Llosa

2020. Toward an integrative framework for understanding multimodal L2 writing in the content areas. Journal of Second Language Writing 47 ► pp. 100711 ff.

Gruba, Paul

2020. What Does Language Testing Have to Offer to Multimodal Listening?. In Another Generation of Fundamental Considerations in Language Assessment, ► pp. 43 ff.

Gruba, Paul

2025. Blended Learning. In The Cambridge Handbook of Technology in Language Teaching and Learning, ► pp. 75 ff.

Matsumoto, Yumi & Suresh Canagarajah

2020. The use of gesture, gesture hold, and gaze in trouble-in-talk among multilingual interlocutors in an English as a lingua franca context. Journal of Pragmatics 169 ► pp. 245 ff.

May, Laura, Thomas Crisp, Gary E. Bingham, Renée S. Schwartz, Mario T. Pickens & Kate Woodbridge

2020. The Durable, Dynamic Nature of Genre and Science: A Purpose‐Driven Typology of Science Trade Books. Reading Research Quarterly 55:3 ► pp. 399 ff.

Wei, Li

2020. System remembers Gunther Kress: Gunther Kress’ contribution to the study of language, technology, and learning. System 93 ► pp. 102289 ff.

Canale, Germán

2019. Laptops and Textbooks as Curriculum Artifacts: Audience, Authorization and Ideologies in the Classroom. In Technology, Multimodality and Learning, ► pp. 137 ff.

Canale, Germán

2019. Toward a Multimodal Socio-Semiotic Account of Learning. In Technology, Multimodality and Learning, ► pp. 41 ff.

Ho, Wing Yee Jenifer

2019. Self-Directed Language Learning: A Semiotic Analysis of a Language Learning App. In Virtual Sites as Learning Spaces, ► pp. 295 ff.

Ho, Wing Yee Jenifer

2022. ‘Coming here you should speak Chinese’: the multimodal construction of interculturality in YouTube videos. Language and Intercultural Communication 22:6 ► pp. 662 ff.

Pantaleo, Sylvia

2019. Metalepsis in Elementary Students’ Multimodal Narrative Representations. Research in the Teaching of English 54:1 ► pp. 8 ff.

Pantaleo, Sylvia

2021. The designing of ocean threat comics by elementary students. Multimodal Communication 10:3 ► pp. 229 ff.

Pantaleo, Sylvia

2024. Kindergarten Children’s Talk About Illustration Techniques in an Almost Wordless Picturebook. Early Childhood Education Journal 52:7 ► pp. 1281 ff.

Parkwell, Corina

2019. Emoji as social semiotic resources for meaning-making in discourse: Mapping the functions of the toilet emoji in Cher’s tweets about Donald Trump. Discourse, Context & Media 30 ► pp. 100307 ff.

Ravelli, Louise J.

2019. Multimodality and the register of disciplinary History. Language, Context and Text. The Social Semiotics Forum 1:2 ► pp. 341 ff.

Tan, Lynde, Ching Sing Chai, Feng Deng, Chun Ping Zheng & Nur Arifah Drajati

2019. Examining pre-service teachers’ knowledge of teaching multimodal literacies: a validation of a TPACK survey. Educational Media International 56:4 ► pp. 285 ff.

Wei, Li & Angel M. Y. Lin

2019. Translanguaging classroom discourse: pushing limits, breaking boundaries. Classroom Discourse 10:3-4 ► pp. 209 ff.

Wong, May

2019. Social Semiotics: Setting the Scene. In Multimodal Communication, ► pp. 1 ff.

Wei, Li & Wing Yee (Jenifer) Ho

2018. Language LearningSans Frontiers: A Translanguaging View. Annual Review of Applied Linguistics 38 ► pp. 33 ff.

Hiippala, Tuomo

2017. An overview of research within the Genre and Multimodality framework. Discourse, Context & Media 20 ► pp. 276 ff.

Santos, Marcelo Sousa

2017. A OFERTA DO LETRAMENTO E A GARANTIA DE FUTUROS SOCIAIS: ANÁLISE DAS POLÍTICAS DE LETRAMENTO DA UNESCO E DE SUAS IDEOLOGIAS LINGUÍSTICAS. Trabalhos em Linguística Aplicada 56:2 ► pp. 641 ff.

Arnesen, Thomas, Eyvind Elstad, Gavriel Salomon & Lars Vavik

2016. Educational Technology and Polycontextual Bridging. In Educational Technology and Polycontextual Bridging, ► pp. 3 ff.

De Costa, Peter I., Magda Tigchelaar & Yaqiong Cui

2016. Reflexivity and transnational habitus. AILA Review 29 ► pp. 173 ff.

[no author supplied]

2025. Environments. In The Cambridge Handbook of Technology in Language Teaching and Learning, ► pp. 73 ff.

This list is based on CrossRef data as of 30 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.