Cover not available

Article published In: Sign Language & Linguistics: Online-First Articles

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Italian Sign Language

Searching for common ground between the sociosemiotic perspective and computational annotation

ORCiD logo with link | University of Catania
Published online: 19 December 2025
DOI logo with link
https://doi.org/10.1075/sll.24004.cal
Abstract
This paper presents a protocol for annotating Italian Sign Language (LIS) that merges cognitive and socio-semiotic principles with the technical demands of sign language processing for automatic sign recognition. The protocol is applied to a preliminary LIS dataset and integrates insights from both formalist and functionalist frameworks in sign language processing, aiming to leverage their respective strengths to create a comprehensive and accessible documentation of LIS. Unlike traditional gloss translations, which often introduce ambiguity by not directly associating signs with their meanings and relying on verbal language categories, the proposed annotation model incorporates a multi-layered approach that includes vocal language labels in written Italian and English, information on the Unit of Meaning performed combined with the implementation Typannot, a language-specific system sign languages annotation. This hybrid approach ensures that the annotated data is both human- and computer-readable, enhancing accessibility for both signers and non-signers. The presented multi-layered annotation model not only mitigates ambiguity but also provides a richer, more precise annotation. Initial findings suggest that this protocol can enhance the clarity and usability of annotated data, positioning it as a valuable resource for both linguistic research and technological applications.
Article outline
  • 1.Introduction
  • 2.Developing and applying effective protocols for sign language data computational annotation in a cognitive and socio-semiotic framework
  • 3.Research background
  • 4.Facing challenges and compromise in using glosses for sign language representation
  • 5.Language-specific systems for computational sign language annotation
  • 6.Proposed guidelines for comprehensive sign language annotation
  • 7.An annotation proposal and its application
    • 7.1Collecting the MultiMedaLIS_3 Dataset
    • 7.2Annotating the dataset
      • 7.2.1Tiers
      • 7.2.2Codifying eyes and mouth
      • 7.2.3Tokenization
  • 8.Discussion
  • 9.Conclusions, limitations, and future perspectives
  • Acknowledgements
  • Notes
  • References
References (78)
References
Almasoud, Ameera M. & Hend S. Al-Khalifa. 2011. A proposed semantic machine translation system for translating Arabic text to Arabic Sign Language. Proceedings of the Second Kuwait Conference on e-services and e-systems, 1–6. Google Scholar logo with link to Google Scholar
Antinoro Pizzuto, Elena, Paolo Rossini & Tommaso Russo. 2006. Representing signed languages in written form: questions that need to be posed. 2nd Workshop on the Representation and Processing of Sign Languages (LREC2006), 1–6.Google Scholar logo with link to Google Scholar
Antinoro Pizzuto, Elena, Isabella Chiari & Paolo Rossini. 2008. The representation issue and its multifaceted aspects in constructing sign language corpora: questions, answers, further problems. 3rd Workshop on the Representation and Processing of Sign Languages: Construction and Exploitation of Sign Language Corpora, 150–158.Google Scholar logo with link to Google Scholar
. 2010. Representing signed languages: theoretical, methodological and practical issues. In Massimo Pettorino, Antonella Giannini, Isabela Chiari & Francesca Maria Dovetto (eds.), Spoken communication, 205–240. Cambridge: Cambridge Scholars Publishing.Google Scholar logo with link to Google Scholar
Bates, Elisabeth, Donna Thal & Brian MacWhinney. 1991. A functionalist approach to language and its implications for assessment and intervention. In Tanya M. Gallagher (ed.), Pragmatics of language, 133–161. Berlin: Springer. Google Scholar logo with link to Google Scholar
Bébian, Roch-Ambroise Auguste. 1825. Mimographie, ou essai d’écriture mimique propre a régulariser le langage des Sourds-Muets. Paris: Chez Louis Colas.Google Scholar logo with link to Google Scholar
Bianchini, Claudia S. 2012. Analyse métalinguistique de l’émergence d’un système d’écriture des langues des signes: SignWriting et son application à la langue des signes italienne (LIS). Paris, France: University of Paris 8 PhD dissertation.
2020. How to improve metalinguistic awareness by writing a language without writing: sign languages and SignWriting. Grapholinguistics in the 21st century, 1039–1065.Google Scholar logo with link to Google Scholar
2023. (D)écrire les langues des signes. Approche grapholinguistique des langues des signes. Brest: Fluxus.Google Scholar logo with link to Google Scholar
Bianchini, Claudia S., Dominique Boutet, Adrien Contesse, Claire Danet, Patrick Doan, Thimothée Goguely & Morgane Rébulard. 2017. Écrire les langues des signes: inscription et transcription. 8ème Colloque “Printemps de la typographie 2017: supporter l’écriture”, école Estienne, March 2017, Paris 2, France.Google Scholar logo with link to Google Scholar
Boutet, Dominique. 2018. Pour une approche kinésiologique de la gestualité synthèse. Rouen, France: Université de Rouen-Normandie Habilitation à Diriger des Recherches.Google Scholar logo with link to Google Scholar
Boutet, Dominique, Claudia S. Bianchini, Patrick Doan, Thimothée Goguely, Morgane Rébulard & Claire Danet. 2016. Typannot: a glyphic system for the transcription of handshapes. 7th Conference of the International Society of Gestures Studies “Gesture-Creativity-Multimodality”, July 2016, Paris, France.Google Scholar logo with link to Google Scholar
Caligiore, Gaia. 2024. Codifying the body: exploring the cognitive and socio-semiotic framework in building a multimodal Italian sign language (LIS) dataset. Catania, Italy: University of Catania PhD dissertation.
Cardinaletti, Anna. 2016. Il progetto SpreadTheSign. Blityri. Studi di storia delle idee sui segni e le lingue, 175–182.Google Scholar logo with link to Google Scholar
Cascella, Marco, Michael Rajnik, Abdul Aleem, Scott C. Dulebohn & Raffaella Di Napoli. 2023. Features, evaluation, and treatment of Coronavirus (COVID-19). Treasure Island: StatPearls.Google Scholar logo with link to Google Scholar
Chesi, Cristiano & Carlo Geraci. 2009. Segni al computer. Manuale di documentazione della lingua italiana dei segni e alcune applicazioni computazionali (2010 Edition). Siena: Cantagalli.Google Scholar logo with link to Google Scholar
Cormier, Kearsy, Onno Crasborn & Richard Bank. 2016. Digging into signs: emerging annotation standards for sign language corpora. 7th Workshop on Representation and Processing of Sign Languages: Corpus Mining, 35–40. Portorož, Slovenia.Google Scholar logo with link to Google Scholar
Crasborn, Onno, Richard Bank & Kearsy Cormier. 2015. Digging into signs: towards a gloss annotation standard for sign language corpora. [URL] (Accessed June 10th, 2024).
Crasborn, Onno, Johanna Mesch, Dafydd Waters, Annika Nonhebel, Els van der Kooij, Bencie Woll & Brita Bergman. 2007. Sharing sign language data online: experiences from the ECHO project. International Journal of Corpus Linguistics 12(4). 535–562. Google Scholar logo with link to Google Scholar
Crasborn, Onno & Inge Zwitserlood. 2008. The Corpus NGT: an online corpus for professionals and laymen. 3rd Workshop on the Representation and Processing of Sign Languages, 44–49.Google Scholar logo with link to Google Scholar
Cuxac, Christian. 2000. La langue des signes française (LSF) — Les voies de l’iconicité. Collection Faits de Langues 15–16(98). 56–57.Google Scholar logo with link to Google Scholar
Cuxac, Cristian & Elena Antinoro Pizzuto. 2010. Émergence, norme et variation dans les langues des signes: vers une redéfinition notionnelle. Langage et Société 1311. 37–53. Google Scholar logo with link to Google Scholar
Cuxac, Cristian & Marie Anne Sallandre. 2002. Iconicity in sign language: a theoretical and methodological point of view. Lecture notes in computer science (LNCS) 22981, 173–180. Berlin: Springer.Google Scholar logo with link to Google Scholar
Cuxac, Christian & Marie Anne Sallandre. 2007. Iconicity and arbitrariness in French Sign Language: highly iconic structures, degenerated iconicity and diagrammatic iconicity. Empirical Approaches to Language Typology 36(12). 13–33.Google Scholar logo with link to Google Scholar
Danet, Claire, Thomas Chloé, Adrien Contesse, Morgane Rébulard, Claudia S. Bianchini, Lea Chevrefils & Patrick Doan. 2022. Applying the transcription system Typannot to mouth gestures. Representation and processing of sign languages: multilingual sign language resources, 42–47.Google Scholar logo with link to Google Scholar
de Almeida Freitas, Fernando, Sarajane M. Peres, Otavio de Paula Albuquerque & Marcelo Fantinato. 2023. Leveraging sign language processing with formal SignWriting and deep learning architectures. Brazilian Conference on Intelligent Systems, 299–314. Berlin: Springer. Google Scholar logo with link to Google Scholar
Di Renzo, Alessio, Luca Lamano, Tommaso Lucioli, Barbara Pennacchi, Gabriele Gianfreda, Giulia Pettina, Claudia S. Bianchini, Paolo Rossini & Elena Antinoro Pizzuto. 2011. Scrivere la LIS con il SignWriting. manuale introduttivo. Adattamento alla LIS del manuale di Valerie Sutton “Lessons in SignWriting”. Roma: ISTC-CNR.Google Scholar logo with link to Google Scholar
Doan, Patrick, Dominique Boutet, Adrien Contesse, Claudia S. Bianchini, Claire Danet, Morgane Rébulard, Jean-François Dauphin, Lea Chevrefils, Chloé Thomas & Mathieu Reguer. 2019. Handling sign language handshapes annotation with the typannot typefont. CogniTextes 191. 1–24. Google Scholar logo with link to Google Scholar
Duranti, Alessandro. 2007. Etnopragmatica. La forza nel parlare. Roma: Carocci.Google Scholar logo with link to Google Scholar
. 2014. Traduzione e traducibilità tra lingue dei segni e lingue vocali. Rivista Italiana di Filosofia del Linguaggio 31. 102–111.Google Scholar logo with link to Google Scholar
Fontana, Sabina & Gaia Caligiore. 2021. Italian Sign Language (LIS) and natural language processing: an overview. In Pier Paolo Basile, Valerio Basile, Danilo Croce & Elena Cabrio (eds.), Proceedings of the 4th Workshop on Natural Language for Artificial Intelligence (NL4AI 2020), co-located with the 19th International Conference of the Italian Association for Artificial Intelligence (AIIA 2020).Google Scholar logo with link to Google Scholar
Fontana, Sabina & Erika Raniolo. 2015. Interazioni tra oralità e unità segniche: uno studio sulle labializzazioni nella lingua dei segni italiana (LIS). In G. Schneider, M. Janner & B. Élie (eds.), Proceedings of the VII dies romanicus turicensis, 241–258. Bern: Peter Lang.Google Scholar logo with link to Google Scholar
Fontana, Sabina, Elena Tomasuolo & Gabriele Gianfreda. 2022. Naming in two modalities: manual and mouth gestures in hearing and deaf children. International Journal of Linguistics 14(4). 120–140. Google Scholar logo with link to Google Scholar
Garcia, Brigitte. 2010. Sourds, surdité, langue(s) des signes et épistémiologie des sciences du langage: problématiques de la scripturisation et modélisation des bas niveaux en Langue des Signe Française (LSF). Paris, France: University of Paris 8 Habilitation à diriger des recherches.Google Scholar logo with link to Google Scholar
Geraci, Carlo, Katia Battaglia, Anna Cardinaletti, Carlo Cecchetto, Caterina Donati, Serena Giudice & Emiliano Mereghetti. 2011. The LIS corpus project. Sign Language Studies 11(4). 528–571. Google Scholar logo with link to Google Scholar
Grushkin, Donald A. 2017. Writing signed languages: what for? What form? American Annals of the Deaf 161(5). 509–527. Google Scholar logo with link to Google Scholar
Hanke, Thomas. 2004. HamNoSys — Representing sign language data in language resources and language processing contexts. 4th International Conference on Language Resources and Evaluation, 1–6.Google Scholar logo with link to Google Scholar
Hanke, Thomas, Marc Schulder, Reiner Konrad & Elena Jahn. 2020. Extending the public DGS corpus in size and depth. 9th Workshop on the Representation and Processing of Sign Languages: Sign language resources in the service of the language community, technological challenges and application perspectives, 75–82.Google Scholar logo with link to Google Scholar
Hanke, Thomas & Jakob Storz. 2008. iLex — A database tool for integrating sign language corpus linguistics and sign language lexicography. Proceedings of the 3rd Workshop on the Representation and Processing of Sign Languages: Construction and exploitation of sign language corpora (LREC 2008), 64–67.Google Scholar logo with link to Google Scholar
Hodge, Gabrielle & Onno Crasborn. 2022. Good practices in annotation. In Jordan Fenlon & Julie A. Hochgesang (eds.), Signed language corpora, 46–89. Washington, DC: Gallaudet University Press. Google Scholar logo with link to Google Scholar
Jiang, Zifan, Amit Moryossef, Mathias Müller & Sarah Ebling. 2023. Machine translation between spoken languages and signed languages represented in SignWriting. In Findings of the Association for Computational Linguistics: EACL 2023, 1706–1724, Dubrovnik, Croatia. Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Johnston, Trevor. 2008. Corpus linguistics and signed languages: no lemmata, no corpus. In Onno Crasborn, Thomas Hanke, Ernst D. Thoutenhoofd, Inge Zwitserlood & Eleni Efthimiou (eds.), Construction and exploitation of sign language corpora. Proceedings of the 3rd Workshop on the Representation and Processing of Sign Languages, 82–87.Google Scholar logo with link to Google Scholar
. 2019. Auslan corpus annotation guidelines (August 2019 version).Google Scholar logo with link to Google Scholar
Johnston, Trevor, Jane van Roekel & Adam Schembri. 2015. On the conventionalization of mouth actions in Australian Sign Language. Language and Speech 59(1). 1–40.Google Scholar logo with link to Google Scholar
Kendon, Adam. 2004. Gesture: visible action as utterance. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Kipp, Michael. 2001. ANVIL. A generic annotation tool for multimodal dialogue. Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), 1367–1370. Google Scholar logo with link to Google Scholar
. 2012. Multimedia annotation, querying, and analysis in Anvil. Multimedia information extraction, 351–367. Google Scholar logo with link to Google Scholar
Klima, Edward S. & Ursula Bellugi. 1979. The signs of language. Cambridge, MA: Harvard University Press.Google Scholar logo with link to Google Scholar
Kopf, Maria, Marc Schulder, Thomas Hanke & Sam Bigeard. 2022. Specification for the harmonization of sign language annotations. [URL] (Accessed June 7th, 2024).
Lakoff, George & Mark Johnson. 1980. Metaphors we live by. Chicago: The University of Chicago Press.Google Scholar logo with link to Google Scholar
Lo Re, Luca. 2022. Corpus multimodale dell’Italiano parlato: basi metodologiche per la creazione di un prototipo. Florence, Italy: University of Florence PhD dissertation.
Matsumoto, Tadahiro, Mihoko Kato & Takashi Ikeda. 2008. Jspad: a sign language writing tool using SignWriting. Proceedings of the 3rd International Universal Communication Symposium, 363–367. Google Scholar logo with link to Google Scholar
McElvenny, James. 2019. Form and formalism in linguistics. Berlin: Language Science Press.Google Scholar logo with link to Google Scholar
Mesch, Johanna. 2012. Swedish Sign Language Corpus. Deaf Studies Digital Journal 31.Google Scholar logo with link to Google Scholar
Mesch, Johanna & Lars Wallin. 2015. Gloss annotations in the Swedish Sign Language corpus. International Journal of Corpus Linguistics 201. 102–120. Google Scholar logo with link to Google Scholar
McNeill, David. 2005. Gesture and thought. Chicago: The University of Chicago Press. Google Scholar logo with link to Google Scholar
Moryossef, Amit, Zifan Jiang, Mathias Müller, Sarah Ebling & Yoav Goldberg. 2023. Linguistically motivated sign language segmentation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 12703–12724, Singapore. Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Neidle, Carol & Christian Vogler. 2012. A new web interface to facilitate access to corpora: development of the ASLLRP data access interface. 5th Workshop on the Representation and Processing of Sign Languages: interactions between corpus and lexicon, 1–7.Google Scholar logo with link to Google Scholar
Nonhebel, Annika, Onno Crasborn & Els van der Kooij. 2004. Sign language transcription conventions for the ECHO project. Version 9. [URL] (Accessed June 7th, 2024).
Peters, Cynthia L. 2000. Deaf American literature from carnival to the canon. Washington, DC: Gallaudet University Press.Google Scholar logo with link to Google Scholar
Petitta, Giulia, Alessio Di Renzo, Isabella Chiari & Paolo Rossini. 2013. Sign language representation: new approaches to the study of Italian Sign Language (LIS). In Laurence Meurant, Aurélie Sinte, Mieke Van Herreweghe & Myriam Vermeerbergen (eds.), Sign language research, uses and practices: Crossing the views on theoretical and applied sign language linguistics, 137–158. Berlin & Nijmegen: De Gruyter Mouton & Ishara Press. Google Scholar logo with link to Google Scholar
Prillwitz, Siegmund. 1989. HamNoSys. Version 2.0. Hamburg notation system for sign languages: an introductory guide. Hamburg: Signum Press.Google Scholar logo with link to Google Scholar
Raniolo, Erika & Gaia Caligiore. 2023. Comunicazione tra sordi e udenti in ambito medico: sfide e opportunità per un’assistenza sanitaria inclusiva. Poster presentation at 5° Convegno Nazionale LIS. “La Lingua dei Segni Italiana: patrimonio culturale o capitale umano?”. Rende: University of Calabria.
Riccardo, Flavia, Marco Ajelli, Xanthi D. Andrianou, Antonino Bella, Martina Del Manso, Massimo Fabiani, Stefania Bellino, Stefano Boros, Alberto Mateo Urdiales, Valentina Marziano, Maria Cristina Rota, Antonietta Filia, Fortunato D’Ancona, Andrea Siddu, Ornella Punzo, Filippo Trentini, Giorgio Guzzetta, Piero Poletti, Paola Stefanelli, Maria Rita Castrucci, Alessandro Ciervo, Corrado Di Benedetto, Marco Tallon, Andrea Piccioli, Silvio Brusaferro, Giovanni Rezza, Stefano Merler & Patrizio Pezzotti. 2020. Epidemiological characteristics of COVID-19 cases and estimates of the reproductive numbers 1 month into the epidemic, Italy, 28 January to 31 March 2020. Eurosurveillance 25(49). Google Scholar logo with link to Google Scholar
Romeo, Orazio. 1991. Dizionario dei segni. La lingua dei segni in 1400 immagini. Bologna: Zanichelli.Google Scholar logo with link to Google Scholar
Russo Cardona, Tommaso. 2004. La mappa poggiata sull’isola. Iconicità e metafora nelle lingue dei segni e nelle lingue vocali. Rende: Centro editoriale e librario università degli studi della Calabria.Google Scholar logo with link to Google Scholar
Santoro, Mirko & Fabio Poletti. 2011. L’annotazione del corpus. In Anna Cardinaletti, Carlo Cecchetto & Caterina Donati (eds.), Grammatica, lessico e dimensioni di variazione della LIS, 69–78. Milan: FrancoAngeli.Google Scholar logo with link to Google Scholar
Sallandre, Marie Anne & Brigitte Garcia. 2013. Epistemological issues in the semiological model for the annotation of sign languages. In Laurence Meurant, Aurélie Sinte, Mieke Van Herreweghe & Myriam Vermeerbergen (eds.), Sign language research, uses and practices: Crossing the views on theoretical and applied sign language linguistics, 159–178. Berlin & Nijmegen: De Gruyter Mouton & Ishara Press. Google Scholar logo with link to Google Scholar
Schembri, Adam & Kearsy Cormier. 2022. Signed language corpora: future directions. In Jordan Fenlon & Julie A. Hochgesang (eds.), Signed language corpora, 196–220. Washington, DC: Gallaudet University Press. Google Scholar logo with link to Google Scholar
Schembri, Adam, Jordan Fenlon, Ramas Rentelis & Kearsy Cormier. 2014. British Sign Language Corpus Project: a corpus of digital video data and annotations of British Sign Language 2008–2014 (2nd edition). London: University College London.Google Scholar logo with link to Google Scholar
. 2017. British Sign Language Corpus Project: a corpus of digital video data and annotations of British Sign Language 2008–2017 (3rd edition). London: University College London.Google Scholar logo with link to Google Scholar
Sevilla, Antonio F., Alberto D. Esteban & José Maria Lahoz-Bengoechea. 2023. Automatic SignWriting recognition: combining machine learning and expert knowledge to solve a novel problem. IEEE Access 111, 13211–13222. Google Scholar logo with link to Google Scholar
Sloetjes, Han. 2023. ELAN-Linguistic annotator. [URL] (Accessed April 4, 2024).
Stokoe, William C. 1960. Sign language structure: an outline of the visual communication systems of the American deaf. Studies in Linguistics Occasional Papers 8. Buffalo: University of Buffalo Press [Re-issued 2005, Journal of Deaf Studies and Deaf Education 10(1), 3–37].Google Scholar logo with link to Google Scholar
Sutton, Valerie. 1981. Lessons in SignWriting. Newport Beach, CA: The center for Sutton movement writing.Google Scholar logo with link to Google Scholar
Volterra, Virginia, Maria Roccaforte, Alessio Di Renzo & Sabina Fontana. 2022. Italian Sign Language from a Cognitive and Socio-Semiotic Perspective: Implications for a General Language Theory. Amsterdam: John Benjamins.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue