Cover not available

In:Linguistic Foundations of Narration in Spoken and Sign Languages:
Edited by Annika Hübl and Markus Steinbach
[Linguistik Aktuell/Linguistics Today 247] 2018
► pp. 223–250

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Ways of expressing action in multimodal narrations – the semiotic complexity of character viewpoint depictions

 | Chemnitz University of Technology
 | European University Viadrina
 | European University Viadrina
Published online: 25 May 2018
DOI logo with link
https://doi.org/10.1075/la.247.10bre
Based on an analysis accounting for the whole body as a possible articulator in the depiction of actions, this chapter argues for an expansion of the notion of ‘character viewpoint gestures’ to a notion of ‘multimodal action depiction from a character viewpoint’. Our study shows that speakers may deploy only single articulators, providing a semantically reduced depiction of the action, or they may deploy more bodily articulators and give a semantically rich picture of the event narrated. Our findings suggest a continuum of semiotic complexity, capturing the range of bodily involvement from less pantomimic (single articulators involved) to pantomimic depictions (more articulators involved) of actions. The paper closes by discussing our observations with respect to the notion of ‘constructed action’ and ‘role shift’ in sign languages and by giving some general remarks on the multimodal analysis of narrations.
Article outline
  • 1.Introduction
  • 2.Analyzing multimodal narratives
  • 3.Exemplification of the method
    • Step 1Description of articulators involved in the character viewpoint depiction
    • Step 2Meaning analysis of body movements
      • Step 2.1Analysis of meaning conveyed by each articulator involved in the character-viewpoint depictions
      • Step 2.2Analysis of the meaning in the verbal and interactive context
    • Step 3Determination of degree of semiotic complexity
  • 4.Ways of expression action in multimodal narrations
    • 4.1Character viewpoint depiction involving a single articulator
    • 4.2Character viewpoint depiction involving two articulators
    • 4.3Character viewpoint depiction involving several articulators
  • 5.Continuum of semiotic complexity
  • 6.Concluding thoughts: Iconicity of action depiction and constructed action in gesture and sign language
  • Acknowledgements
  • Notes
  • References
References (94)
References
Beattie, G. & Shovelton, H. 2001. How gesture viewpoint influences what information decoders receive from iconic gestures. In Oralite et gestualite: Interactions et comportements multimodaux dans la communication, C. Cave, I. Guaitella & S. Santi (eds), 283–287. Paris: L’Harmattan.Google Scholar logo with link to Google Scholar
2002. An experimental investigation of some properties of individual iconic gestures that mediate their communicative power. British Journal of Psychology 93(2): 179–192.Google Scholar logo with link to Google Scholar
Birdwhistell, R. 1970. Kinesics and Context. Essays on Body, Motion, Communication. Philadelphia PA: University of Pennsylvania Press.Google Scholar logo with link to Google Scholar
Bressem, J. 2013. A linguistic perspective on the notation of form features in gestures. In C. Müller et al. (eds), 1079–1098.Google Scholar logo with link to Google Scholar
2015. Repetition als Mittel der Musterbildung bei redebegleitenden Gesten. In Handbuch Satz, Äußerung, Schema, C. Dürscheid & J. G. Schneider (eds), 422–442. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Bressem, J., Ladewig, S. H. & Müller, C. 2013. Linguistic Annotation System for Gestures (LASG). In Müller et al.. (eds), 1098–1125.Google Scholar logo with link to Google Scholar
Bressem, J. & Ladewig, S. H. 2011. Rethinking gesture phases – Articulatory features of gestural movement? Semiotica 184(1–4): 53–91.Google Scholar logo with link to Google Scholar
Bressem, J. & Müller, C. 2014. The family of AWAY-gestures: Negation, refusal and negative assessement. In Müller et al.. (eds), 1592–1605.Google Scholar logo with link to Google Scholar
Bühler, K. 1934. Sprachtheorie. Die Darstellungsfunktion der Sprache. Stuttgart: Fischer.Google Scholar logo with link to Google Scholar
Casey, S. & Emmorey, K. 2009. Co-speech gesture in bimodal bilinguals. Language and Cognitive Processes 24(2): 290–312.Google Scholar logo with link to Google Scholar
Cassell, J. & McNeill, D. 1991. Gesture and the poetics of prose. Poetics Today 12(3): 375–404.Google Scholar logo with link to Google Scholar
Cormier, K., Quinto-Pozos, D., Sevcikova, Z. & Schembri, A. 2012. Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language and Communication 32: 329–348.Google Scholar logo with link to Google Scholar
Davis, M. 1979. Laban analysis of nonverbal communication. In Nonverbal Communication: Readings with Commentary, S. Weitz (ed.), 182–206. Oxford: OUP.Google Scholar logo with link to Google Scholar
Dudis, P. G. 2004. Body partitioning and real-space blends. Cognitive Linguistics 15(2): 223–238.Google Scholar logo with link to Google Scholar
Dudis, P. G. 2007. Types of depiction in ASL. Ms, Gallaudet University. <[URL]>Google Scholar logo with link to Google Scholar
Duncan, S. 2005. Gesture in signing: A case study in Taiwan Sign Language. Language and Linguistics 6(2): 279–318.Google Scholar logo with link to Google Scholar
Eibl-Eibesfeldt, I. 1971. Transcultural patterns of ritualized contact behavior. In Behavior and Environment. The Use of Space by Animals and Men, A. H. Esser (ed.), 238–246. New York NY: Plenum.Google Scholar logo with link to Google Scholar
Ekman, P., Sorenson, E. R. & Friesen, W. 1969. Pan-cultural elements in facial displays of emotions. Science 164(3875): 86–88.Google Scholar logo with link to Google Scholar
Ekman, P., Friesen, W. & Hager, J. C. 2002. Facial Action Coding System. Salt Lake City UT: Research Nexus.Google Scholar logo with link to Google Scholar
Frey, S., Hirsbrunner, H. P., Pool, J. & Daw, W. 1981. Das Berner System zur Untersuchung nonverbaler Interaktion: I. Die Erhebung des Rohdatenprotokolls; II. Die Auswertung von Zeitreihen visuell-auditiver Information. In Methoden der Analyse von Face-to-Face-Situationen, P. Winkler (ed.), 203–268. Stuttgart: Metzler.Google Scholar logo with link to Google Scholar
Goodwin, C. 1981. Conversational organization: Interaction between speakers and hearers. New York: Academic Press New York.Google Scholar logo with link to Google Scholar
Gumperz, J. 1992. Contextualization revisited. In The Contextualization of Language [Pragmatics & Beyond New Series 22], P. Auer & A. di Luzio (eds), 39–54. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Gülich, E. 1970. Makrosyntax der Gliederungssignale im gesprochenen Französisch. Munich: Fink.Google Scholar logo with link to Google Scholar
Gülich, E. & Hausendorf, H. 2000. Vertextungsmuster Narration. In Text-und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 369–385. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Günthner, S. 1996. The prosodic contextualization of moral work: An analysis of reproaches in ‘why’-formats. In Prosody in Conversation, E. Couper-Kuhlen & M. Selting (eds), 271–302. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Harrison, S. 2009. The expression of negation through grammar and gesture. In Studies in Language and Cognition, J. Zlatev, M. Andren, M. J. Falck & C. Lundmark (eds), 405–409. Cambridge: CUP.Google Scholar logo with link to Google Scholar
2014. Head shakes: Variation in form, function, and cultural distribution of a head movement related to “no”. In C. Müller et al.. (eds), 1496–1501.Google Scholar logo with link to Google Scholar
Hermann, A. & Pendzich, N.-K. 2014. Nonmanual gestures in sign languages. In C. Müller et al.. (eds), 2149–2163.Google Scholar logo with link to Google Scholar
Herrmann, A. & Steinbach, M. 2012. Quotation in sign languages. In Quotatives: Cross-linguistic and Cross-disciplinary Perspectives [Converging Evidence in Language and Communication Research 15], I. van Alphen & I. Buchstaller (eds), 203–228. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Jakobson, R. 1966. Selected writings. Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Keim, I. 1993. Sprachvariation als konstitutives Merkmal eines sozialen Stils. Am Beispiel einer innerstädtischen Welt „Kleiner Leute“. Soziolinguistik. Der Deutschunterricht 3: 44–56.Google Scholar logo with link to Google Scholar
Kendon, A. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63.Google Scholar logo with link to Google Scholar
1980. Gesture and speech: Two aspects of the process of utterance. In Nonverbal Communication and Language, M. R. Key (ed.), 207–227. The Hague: Mouton.Google Scholar logo with link to Google Scholar
1992. Some recent work from Italy on quotable gestures (emblems). Journal of Linguistic Anthropology 2(1): 92–108.Google Scholar logo with link to Google Scholar
2000. Language and gesture: Unity or duality. In Language and Gesture, D. McNeill (ed.), 47–63. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Kendon, A. 2004. Gesture: Visible Action as Utterance. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Kendon, A. & Ferber, A. 1973. A description of some human greetings. In Comparative Ecology and Behavior of Primates, R. P. Michael & J. H. Crook (eds), 591–668. New York NY: Academic Press.Google Scholar logo with link to Google Scholar
Kita, S. & Özyürek, A. 2003. What does cross-linguistic variation in semantic coordination of speech and gesture reveal? Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48: 16–32.Google Scholar logo with link to Google Scholar
Knapp, M., Hall, J. & Horgan, T. 2013. Nonverbal Communication in Human Interaction. Boston MA: Wadsworth Cengage Learning.Google Scholar logo with link to Google Scholar
Ladewig, S. H. 2010. Beschreiben, suchen und auffordern – Varianten einer rekurrenten Geste. Sprache und Literatur 41(105): 89–111.Google Scholar logo with link to Google Scholar
2011. Putting the cyclic gesture on a cognitive basis. CogniTextes 6. <[URL]> (June 2013).Google Scholar logo with link to Google Scholar
2014c. Creating multimodal utterance. The linear integration of gestures into speech. In: Müller, C. et al. (eds), 1662–1677.Google Scholar logo with link to Google Scholar
Ladewig, S. H. & Bressem, J. 2013. New insights into the medium hand – Discovering structures in gestures based on the four parameters of sign language. Semiotica 197: 203–231.Google Scholar logo with link to Google Scholar
Ladewig, S. H. (forthcoming). Integrating Gestures – cognitive grammar multimodal. Berlin: De Gruyter Mouton.
Ladewig, S. H. 2014a. Recurrent gestures. In C. Müller et al.. (eds), 1558–1575.Google Scholar logo with link to Google Scholar
. 2014b. The cyclic gesture. In C. Müller et al.. (eds), 1605–1618.Google Scholar logo with link to Google Scholar
Langacker, R. 2008. Cognitive Grammar: A Basic Introduction. Oxford: OUP.Google Scholar logo with link to Google Scholar
Lausberg, H. & Sloetjes, H. 2009. Coding gestural behavior with the NEUROGES – ELAN system. Behavioral Research Methods 41(3): 841–849.Google Scholar logo with link to Google Scholar
van Loon, E., Pfau, R. & Steinbach, M. 2014. The grammaticalization of gestures in sign languages. In C. Müller et al. (eds), 2133–2149.Google Scholar logo with link to Google Scholar
Mandel, M. 1977. Iconic devices in American Sign Language. In On the other hand. New perspectives on American Sign Language, L. A. Friedman (ed.) New York: Academic PressGoogle Scholar logo with link to Google Scholar
Mayerthaler, W. 1980. Ikonismus in der Morphologie. Zeitschrift für Semiotik 2: 19–37.Google Scholar logo with link to Google Scholar
McClave, E. Z. 2000. Linguistic functions of head movements in the context of speech. Journal of Pragmatics 32: 855–878.Google Scholar logo with link to Google Scholar
McNeill, D. 1992. Hand and Mind. What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
2005. Gesture and Thought. Chicago IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
Mittelberg, I. 2006. Metaphor and Metonymy in Language and Gesture: Discoursive Evidence for Multimodal Models of Grammar. PhD dissertation, Cornell University. Ann Arbor MI: ProQuest/UMI. <[URL]>Google Scholar logo with link to Google Scholar
2014. Gestures and iconicity. In C. Müller et al.. (eds), 1712–1732.Google Scholar logo with link to Google Scholar
Mittelberg, I. & Waugh, L. 2014. Gestures and metonymy. In C. Müller et al.. (eds), 1747–1766.Google Scholar logo with link to Google Scholar
Müller, C. 1998. Redebegleitende Gesten: Kulturgeschichte, Theorie, Sprachvergleich. Berlin: Arno Spitz.Google Scholar logo with link to Google Scholar
2004. Forms and uses of the Palm Up Open Hand. A case of a gesture family? In Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 233–256.Berlin: Weidler VerlagGoogle Scholar logo with link to Google Scholar
Müller, C. 2008. Metaphors. Dead and Alive, Sleeping and Waking. A Dynamic View. Chicago, University of Chicago Press.Google Scholar logo with link to Google Scholar
Müller, C. 2010. Wie Gesten bedeuten. Eine kognitiv-linguistische und sequenzanalytische Perspektive. Sprache und Literatur 41(105): 37–68.Google Scholar logo with link to Google Scholar
2013. Gestures as a medium of expression: The linguistic potential of gestures. In C. Müller et al.. (eds), 202–217.Google Scholar logo with link to Google Scholar
2014a. Gestural modes of representation as techniques of depiction. In Müller et al.. (eds), 1677–1687. Berlin: De Gruyter Mouton.Google Scholar logo with link to Google Scholar
2014b. Gesture as “deliberate expressive movement.” In From Gesture in Conversation to Visible Action as Utterance: Essays in Honor of Adam Kendon, M. Seyfeddinipur & M. Gullberg (eds), 127–152. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Müller, C., Bressem, J. & Ladewig S. H. 2013. Towards a grammar of gesture: A form-based view. In Müller et al.. (eds), 707–733.Google Scholar logo with link to Google Scholar
C. Müller, C., Ladewig, S. H. & Bressem, J. 2013. Gestures and speech from a linguistic perspective: A new field and its history. In C. Müller et al.. (eds), 55–81.Google Scholar logo with link to Google Scholar
Müller, C., Ladewig S. H. & Teßendorf S. 2010. Structures of meaning and reference – Investigating gestures of the concrete and abstract. Paper presented at the 4th Conference of the International Society for Gesture Studies, Frankfurt/Oder, Germany.Google Scholar logo with link to Google Scholar
Müller, C. & Tag, S. 2010. The dynamics of metaphor: Foregrounding and activating metaphoricity in conversational interaction. Cognitive Semiotics 6: 85–120.Google Scholar logo with link to Google Scholar
Müller, C., Cienki, A, Fricke, E., Ladewig, S. H., McNeill, D. & Teßendorf, S. (eds). 2013. Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 1. Berlin, Boston: De Gruyter Mouton.Google Scholar logo with link to Google Scholar
Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds). 2014. Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 2. Berlin, Boston: De Gruyter: Mouton.Google Scholar logo with link to Google Scholar
2010. Viewpoint in speech-gesture integration: Linguistic structure, discourse structure, and event structure. Language and Cognitive Processes 25(5): 650–668.Google Scholar logo with link to Google Scholar
Peirce, C. S. 1960. Collected Papers of Charles Sanders Peirce (1931–1958), Vol. I: Principles of Philosophy, Vol. II: Elements of Logic , C. Hartshorne & P. Weiss (eds). Cambridge MA: The Belknap of Harvard University Press.Google Scholar logo with link to Google Scholar
Pfau, R. & Steinbach, M. 2006. Modality-independent and modality specific aspects of grammaticalization in sign languages. Linguistics in Potsdam 24: 3–98. <[URL]>Google Scholar logo with link to Google Scholar
Pörings, R. & Schmitz, U. 2003. Sprache und Sprachwissenschaft. Eine kognitiv orientierte Einführung. Tübingen: Gunter Narr.Google Scholar logo with link to Google Scholar
Quasthoff, U. 1979. Konversationelle Erzählungen im Sozialamt: Ihr Einsatz als kommunikative Strategien. In Soziologische Analysen. Referate aus den Veranstaltungen der Sektionen der deutschen Gesellschaft für Soziologie und der ad-hoc-Gruppen beim 19. Deutschen Soziologentag, R. Mackensen & F. Sagebiel (eds), 515–534. Berlin: Deutsche Gesellschaft für Soziologie (DGS).Google Scholar logo with link to Google Scholar
2000. Erzählen als interaktive Gesprächsstruktur. In Text- und Gesprächslinguistik/Linguistics of Text and Conversation, K. Brinker, G. Antos, W. Heinemann & S. F. Sager (eds), 1293–1309. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Quinto-Pozos, D. 2007. Why does constructed action seem obligatory? An analysis of “classifiers” and the lack of articulator-referent correspondence. Sign Language Studies 7(4): 458.Google Scholar logo with link to Google Scholar
2014. Enactment as a (signed) language communicative strategy. In C. Müller et al.. (eds), 2163–2170.Google Scholar logo with link to Google Scholar
Quinto-Pozos, D. & Parrill, F. 2012. Comparing viewpoint strategies used by co-speech gesturers and signers. Paper presented at the 5th Conference of the International Society for Gesture Studies, Lund, Sweden.Google Scholar logo with link to Google Scholar
Selting, M. 1995. Sprechstile als Kontextualisierungshinweise. In Stilfragen. Jahrbuch 1994 des Instituts für deutsche Sprache, G. Stickel (ed.), 225–256. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Selting, M., Auer, P., Barth-Weingarten, D., Bergmann, J., Bergmann, P., Birkner, K., Couper-Kuhlen, E., Deppermann, E., Gilles, P., Günther, S., Hartung, M., Kern, F., Mertzlufft, C., Meyer, C., Morek, M., Oberzaucher, F., Peter, L., Quasthoff, U. M., Schütte, W., Stukenbrock A. & Uhmann, S. 2009. Gesprächsanalytisches Transkriptionssystem 2 (GAT 2). Gesprächsforschung – Online Zeitschrift zur verbalen Interaktion 10: 353–402.Google Scholar logo with link to Google Scholar
Seyfeddinipur, M. 2004. Meta-discursive gestures from Iran: Some uses of the ‘Pistol Hand’. In The Semantics and Pragmatics of Everyday Gestures, C. Müller & R. Posner (eds), 205–216. Berlin: Weidler.Google Scholar logo with link to Google Scholar
Sweetser, E. 2012. Introduction: Viewpoint and perspective in language and gesture, from the Ground down. In Viewpoint in Language: A Multimodal Perspective, B. Dancygier & E. Sweetser (eds), 1–22. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Sweetser, E. & Stec, K. 2013. Managing multiple viewpoints: Coordinating embedded perspective in multimodal narratives. Paper presented at the ICLC-12, Edmonton, Canada.Google Scholar logo with link to Google Scholar
Taub, S. 2001. Language from the Body: Iconicity and Metaphor in American Sign Language. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Teßendorf, S. 2014. Pragmatic and metaphoric gestures – Combining functional with cognitive approaches. In C. Müller et al.. (eds), 1540–1558.Google Scholar logo with link to Google Scholar
2016. Actions as sources of gestures. In Beyond Language Boundaries: Multimodal Use in Multilingual Contexts, M. Fernández-Villanueva & K. Jungbluth (eds), 34–54. Berlin, Boston: De Gruyter Google Scholar logo with link to Google Scholar
2007. Routes from gesture to language. In Verbal and Signed Languages: Comparing Structures, Constructs and Methodologies, E. Pizzuto, P. Pientrandrea & R. Simone (eds), 107–137. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Wilcox, S., Rossini P. & Antinoro E. P. 2010. Grammaticalization in sign languages. In Sign Languages, D. Brentari (ed.), 332–355. Cambridge: CUP.Google Scholar logo with link to Google Scholar
Wilcox, S. & Xavier A. N. 2013. A framework for unifying spoken language, signed language, and gesture. Todas as Letras-Revista de Língua e Literatura 15(1): 88–110.Google Scholar logo with link to Google Scholar
Cited by (12)

Cited by 12 other publications

Pfänder, Stefan, Elisabeth Gülich, Sibylle Kriegel, Elke Schumann & Daniel Mandel
2025. Redewiedergabe in retellings: Emergenz und Modell. Romanistisches Jahrbuch 76:1 ► pp. 191 ff. DOI logo
Zhang, Icy (Yunyi), Tina Izad & Erica A. Cartmill
2024. Embodying Similarity and Difference: The Effect of Listing and Contrasting Gestures During U.S. Political Speech. Cognitive Science 48:3 DOI logo
Bellifemine, Corrado & Camille Dupret
2023. Référence multimodale dans les narrations d’enfants : les gestes servent-ils à clarifier les expressions référentielles ambiguës ?. Discours :32 DOI logo
Gabarró-López, Sílvia & Laurence Meurant
2022. Contrasting signed and spoken languages. Languages in Contrast 22:2 ► pp. 169 ff. DOI logo
Ingram, Matthew Bruce & Ian Mitchell Wallace
2022. Making a mark: Transforming everyday work of city arborists into dance. Frontiers in Communication 7 DOI logo
Quinto-Pozos, David, Fey Parrill & Caitie Coons
2022. The interface between grammar and bodily enactment in ASL and English. Languages in Contrast 22:2 ► pp. 195 ff. DOI logo
Vandenitte, Sébastien
2022. Making Referents Seen and Heard Across Signed and Spoken Languages: Documenting and Interpreting Cross-Modal Differences in the Use of Enactment. Frontiers in Psychology 13 DOI logo
Savchuk, Svetlana
2021. On the diachronic aspect of gestural communication. Languages and Modalities 1 ► pp. 95 ff. DOI logo
Katila, Julia & Johanne S. Philipsen
2019. The intercorporeality of closing a curtain. Pragmatics & Cognition 26:2-3 ► pp. 167 ff. DOI logo
Philipsen, Johanne S. & Sarah Bro Trasmundi
2019. Gesture reuse as distributed embodied cognition. Gesture 18:1 ► pp. 1 ff. DOI logo
Müller, Cornelia
2018. Gesture and Sign: Cataclysmic Break or Dynamic Relations?. Frontiers in Psychology 9 DOI logo
Müller, Cornelia
2024. A Toolbox of Methods for Gesture Analysis. In The Cambridge Handbook of Gesture Studies, ► pp. 182 ff. DOI logo

This list is based on CrossRef data as of 22 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue