Cover not available

In:Advances in Sign Language Corpus Linguistics
Edited by Ella Wehrmeyer
[Studies in Corpus Linguistics 108] 2023
► pp. 353–382

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 13
Utterance unit annotation for the Japanese Sign Language Dialogue Corpus

Towards a method for detecting interactional boundaries in spontaneous sign language dialogue

 | National Institute of Informatics, Japan | The Graduate University for Advanced Studies, SOKENDAI, Japan
 | The Graduate University for Advanced Studies, SOKENDAI, Japan
 | Tsukuba University of Technology, Japan
 | Future University Hakodate, Japan
 | The Graduate University for Advanced Studies, SOKENDAI, Japan
 | The University of Tokyo, Japan
 | Tsukuba University of Technology, Japan
Published online: 3 April 2023
DOI logo with link
https://doi.org/10.1075/scl.108.13bon
Abstract
This chapter defines ‘utterance units’ and describes their annotation in the Japanese Sign Language (JSL) dialogue corpus, first focusing on how human annotators – native signers of JSL – identify and annotate utterance units, before reporting on part of speech (POS) tagging for JSL and semi-automatic annotation of utterance units. The utterance unit is an original concept for segmenting and annotating movement features in sign language dialogue, based on signers’ native sense. We postulate a fundamental interaction-specific unit for understanding interactional mechanisms (such as turn-taking) in sign language social interactions from the perspectives of conversation analysis and multimodal interaction studies. We explain differences between sentence and utterance units, the corpus construction and composition, and the annotation scheme, before analyzing how JSL native annotators annotated the units. Finally, we show the application potential of this research by presenting two case studies, the first exploring POS annotations, and the second a first attempt at automatic annotation using OpenPose software.
Article outline
  • 1.Introduction
  • 2.Sentence versus utterance units
  • 3.The Colloquial Corpus of Japanese Sign Language
    • 3.1Tasks, areas and participants
    • 3.2Filming services and video clip editing
  • 4.Annotation of utterance units
    • 4.1Identifying an utterance unit
    • 4.2Annotation of utterance units on the individual level
    • 4.3Integration level
  • 5.Trial annotation of utterance units
    • 5.1Quantitative analysis
    • 5.2Qualitative analysis of utterance units
      • 5.2.1Utterance unit including mouthing
      • 5.2.2Utterance unit segmented by gaze shift
      • 5.2.3Utterance unit bounded by interlocutor’s actions
  • 6.The application potential of this research
    • 6.1Part of speech annotation for the utterance unit
      • 6.1.1Target data and annotation tool
      • 6.1.2POS annotation guidelines
      • 6.1.3Results, challenges and discussions
    • 6.2Automatic detection of utterance units
      • 6.2.1Detecting body-keypoint positions
      • 6.2.2Results
      • 6.2.2Discussion
  • 7.Conclusions
  • Notes
  • Acknowledgments
  • References
  • Appendix
References (38)
References
Bank, Richard, Crasborn, Onno & Van Hout, Roeland. 2016. The prominence of spoken language elements in a sign language. Linguistics 54(6): 1281–1305. Google Scholar logo with link to Google Scholar
Bono, Mayumi, Kikuchi, Kouhei, Cibulka, Paul & Osugi, Yutaka. 2014. A colloquial corpus of Japanese Sign Language: A design of language resources for observing sign language conversations. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC 2014), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard … Stelios Piperidis (eds), 1898–1904. Paris: ELRA. <[URL]> (27 August 2022).
Bono, Mayumi, Sakaida, Rui, Okada, Tomohiro, & Miyao, Yusuke. 2020. Utterance-unit annotation for the JSL Dialogue Corpus: Toward a multimodal approach to corpus linguistics. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie Hochgesang, Jette Kristofferson & Johanna Mesch (eds), 13–20. Paris: ELRA. <[URL]> (27 August 2022).
Boyes Braem, Penny & Sutton-Spence, Rachel (eds). 2001. The Hands are the Head of the Mouth: The Mouth as Articulator in Sign Languages. Berlin: Signum.Google Scholar logo with link to Google Scholar
Cao, Zhe, Simon, Tomas, Wei, Shih-En & Sheikh, Yaser. 2017. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 1302–1310. Google Scholar logo with link to Google Scholar
Cassidy, Stephen, Crasborn, Onno, Nieminen, Henri, Stoop, Wessel, Hulsbosch, Micha, Even, Susan, Komen, Erwin & Johnston, Trevor. 2018. Signbank: Software to support web based dictionaries of sign language. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC 2018), Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi … Takenobu Tokunaga (eds), 2359–2364. Paris: ELRA. <[URL]> (27 August 2022).
Celebi, Sait, Aydin, Ali, Temiz, Talha & Arici, Tarik. 2013. Gesture recognition using skeleton data with weighted dynamic time warping. In Proceedings of the International Conference on Computer Vision Theory and Applications, Volume 1 (VISAPP2013), Sebastiano Battiato & José Braz (eds), 620–625. Institute for Systems and Technologies of Information, Control and Communication. Google Scholar logo with link to Google Scholar
Cormier, Kearsy, Quinto-Pozos, David, Sevcikova, Zed & Schembri, Adam. 2012. Lexicalisation and de-lexicalisation processes in sign languages: Comparing depicting constructions and viewpoint gestures. Language & Communication 32(4): 329–348. Google Scholar logo with link to Google Scholar
Cormier, Kearsy, Smith, Sandra & Zwets, Martine. 2013. Framing constructed action in British Sign Language narratives. Journal of Pragmatics 55: 119–139. Google Scholar logo with link to Google Scholar
Crasborn, Onno A. 2007. How to recognize a sentence when you see one. Sign Language & Linguistics 10(2): 103–111. Google Scholar logo with link to Google Scholar
Crasborn, Onno & Sloetjes, Han. 2008. Enhanced ELAN functionality for sign language corpora. In Proceedings of the 3rd Workshop on the Representation and Processing of Sign Languages: Construction and Exploitation of Sign Language Corpora (LREC 2008), Onno Crasborn, Thomas Hanke, Eleni Efthimiou, Inge Zwitserlood & Ernst Thoutenhoofd (eds), 39–43. Paris: ELDA. <[URL]> (27 August 2022).
Crasborn, Onno, Zwitserlood, Inge, Van der Kooij, Els & Ormel, Ellen. 2020. Global Signbank manual, version 2, Form–meaning units in sign languages: An inventory and studies of interpretation and use in Sign Language of the Netherlands (NGT). Google Scholar logo with link to Google Scholar
Den, Yasuharu, Koiso, Hanae, Maruyama, Takehiko, Maekawa, Kikuo, Takanashi, Katsuya, Enomoto, Mika & Yoshida, Nao. 2010. Two-level annotation of utterance units in Japanese dialogs: An empirically emerged scheme. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC 2010), Nicoletta Calzolari, Khalid Choukri, Bente Maegaard, Joseph Mariani, Jan Odijk … Daniel Tapias (eds), 2103–2110. Paris: ELRA. <[URL]> (28 August 2022).
Ford, Cecilia & Thompson, Sandra. 1996. Interactional units in conversation: Syntactic, intonational, and pragmatic resources for the management of turns. In Interaction and Grammar [Studies in Interactional Sociolinguistics 13], Elinor Ochs, Emanuel Schegloff & Sandra Thompson, (eds), 134–184. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Fragkiadakis, Manolis, Nyst, Victoria & Van der Putten, Peter. 2020. Signing as input for a dictionary query: Matching signs based on joint positions of the dominant hand. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives (LREC 2020), Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen & Johanna Mesch (eds), 69–74. Paris: ELRA. <[URL]> (30 August 2022).
Johnston, Trevor, Van Roekel, Jane & Schembri, Adam. 2015. On the conventionalization of mouth actions in Australian Sign Language. Language and Speech 59(1): 1–40. Google Scholar logo with link to Google Scholar
Kendon, Adam. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26: 22–63. Google Scholar logo with link to Google Scholar
. 1970. Movement coordination in social interaction: Some examples described. Acta Psychologica 32(2): 101–125. Google Scholar logo with link to Google Scholar
. 2004. Gesture: Visible Action as Utterance. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Kendrick, Kobin. 2015. Other-initiated repair in English. Open Linguistics 1: 164–190. Google Scholar logo with link to Google Scholar
Kita, Sotaro & Ide, Sachiko. 2007. Nodding, aizuchi, and final particles in Japanese conversation: How conversation reflects the ideology of communication and social relationships. Journal of Pragmatics 39: 1242–1254. Google Scholar logo with link to Google Scholar
Kitzinger, Celia. 2013. Repair. In The Handbook of Conversation Analysis, Jack Sidnell & Tanya Stivers (eds), 229–256. Malden MA: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Levinson, Stephen. 1983. Pragmatics [Cambridge Textbooks in Linguistics Series]. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Liddell, Scott. 2003. Grammar, Gesture, and Meaning in American Sign Language. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Maruyama, Takehiko, Den, Yasuharu & Koiso, Hanae. 2020. Design and annotation of two-level utterance units in Japanese. In In Search of Basic Units of Spoken Language: A Corpus-Driven Approach [Studies in Corpus Linguistics 94], Shlomo Izre’el, Heliana Mello, Alessandro Panunzi & Tommaso Raso (eds), 155–180. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
McNeill, David. 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
. 2005. Gesture and Thought. Chicago IL: University of Chicago Press. Google Scholar logo with link to Google Scholar
Mondada, Lorenza. 2018. Multiple temporalities of language and body in interaction: Challenges for transcribing multimodality. Research on Language and Social Interaction 51: 85–106. Google Scholar logo with link to Google Scholar
Nivre, Joakim, De Marneffe, Marie-Catherine, Ginter, Filip, Hajič, Jan, Manning, Christopher, Pyysalo, Sampo, Schuster, Sebastian, Tyers, Francis & Zeman, Daniel. 2020. Universal Dependencies v2: An ever-growing multilingual treebank collection. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC 2020), Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri … Stelios Piperidis (eds), 4034–4043. Paris: ELRA. <[URL]> (30 August 2022).
Östling, Robert, Börstell, Carl, Gärdenfors, Moa & Wirén, Mats. 2017. Universal Dependencies for Swedish Sign Language. In Proceedings of the 21st Nordic Conference on Computational Linguistics, Jörg Tiedemann & Nina Tahmasebi (eds), 303–308. <[URL]> (30 August 2022).
Osugi, Yutaka & Bono, Mayumi. 2015. Shuwa jinbungaku no kouchiku ni mukete (2): Shuwa gengo co-pasu project (Towards the construction of a humanities of sign language (2): Sign Language corpus project). In Shuwa, gengo, comyunike-shon, Nihon Shuwa Kenkyusho (ed.), 99–136. Kyoto: Bunrikaku.Google Scholar logo with link to Google Scholar
Sacks, Harvey, Schegloff, Emanuel & Jefferson, Gail. 1974. A simplest systematics for the organization of turn-taking for conversation. Language 50: 696–735. Google Scholar logo with link to Google Scholar
Schegloff, Emanuel, Jefferson, Gail & Sacks, Harvey. 1977. The preference for self-correction in the organization of repair in conversation. Language 53(2): 361–382. Google Scholar logo with link to Google Scholar
Schegloff, Emanuel A. 2007. Sequence Organization in Interaction: A Primer in Conversation Analysis. Cambridge: CUP. Google Scholar logo with link to Google Scholar
Schneider, Pascal, Memmesheimer, Raphael, Kramer, Ivanna & Paulus, Dietrich. 2019. Gesture recognition in RGB videos using human body keypoints and dynamic time warping. In RoboCup 2019: Robot World Cup XXIII [Lecture Notes in Computer Science 11531], Stephan Chalup, Tim Niemueller, Jackrit Suthakorn & Mary-Anne Williams (eds), 281–293. Singapore: Springer. Google Scholar logo with link to Google Scholar
Selting, Margret. 2000. The construction of units in conversational talk. Language in Society 29(4): 477–517. Google Scholar logo with link to Google Scholar
Sidnell, Jack & Stivers, Tanya (eds). 2013. The Handbook of Conversation Analysis. Malden MA: Wiley Blackwell. Google Scholar logo with link to Google Scholar
Torigoe, Takashi. 1997. How does a conversation proceed in Japanese Sign Language? Focusing on overlapping of utterance Deaf dyads. Japanese Journal of Sign Linguistics 14(1): 13–20.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue