Cover not available

Article published In: International Journal of Corpus Linguistics
Vol. 28:3 (2023) ► pp.430–459

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Annotation uncertainty in the context of grammatical change

 | University of Zurich
ORCiD logo with link | LMU Munich
ORCiD logo with link | Universität der Bundeswehr München
 | Paderborn University
ORCiD logo with link | LMU Munich
Published online: 20 June 2023
DOI logo with link
https://doi.org/10.1075/ijcl.20113.mer
Abstract
This paper elaborates on the notion of uncertainty in the context of annotation in large text corpora, specifically focusing on (but not limited to) historical languages. Such uncertainty might be due to inherent properties of the language, for example, linguistic ambiguity and overlapping categories of linguistic description, but could also be caused by a lack of annotation expertise. By examining annotation uncertainty in more detail, we identify the sources, deepen our understanding of the nature and different types of uncertainty encountered in daily annotation practice, and discuss practical implications of our theoretical findings. This paper can be seen as an attempt to reconcile the perspectives of the main scientific disciplines involved in corpus projects, linguistics and computer science, to develop a unified view and to highlight the potential synergies between these disciplines.
Article outline
  • 1.Introduction
  • 2.Current annotation practice and limitations
  • 3.Uncertainty in historical (Corpus) linguistics
    • 3.1Project context and underlying corpus
    • 3.2Annotation uncertainties
      • 3.2.1Overlapping categories and the gradualness of change
      • 3.2.2Types of categorical gradience
      • 3.2.3The human annotator as a source of (subjective) uncertainty
  • 4.Mathematical modeling of uncertainty
    • 4.1Frame of discernment and ground-truth
    • 4.2Uncertainty measures and calculi
    • 4.3Vagueness, fuzziness, and graded notions of truth
    • 4.4Granularity
  • 5.A unified view of uncertainty
    • 5.1Fuzziness and ambiguity
    • 5.2Incompleteness and lack of knowledge
  • 6.Practical implications
    • 6.1Types of annotation
    • 6.2Experience from the annotation practice: Tagging ambiguities and uncertainties
      • 6.2.1Human expert annotator
      • 6.2.2Machine annotator
  • 7.Conclusions
  • Notes
  • References
References (48)
References
Aarts, B. (2007). Syntactic Gradience: The Nature of Grammatical Indeterminacy. Cambridge University Press.Google Scholar logo with link to Google Scholar
Russell, B. (1923). Vagueness. The Australasian Journal of Psychology and Philosophy, 11, 84–92. Google Scholar logo with link to Google Scholar
Bley-Vroman, R. (1983). The comparative fallacy in interlanguage studies: The case of systematicity. Language learning, 331, 1–17. Google Scholar logo with link to Google Scholar
Bybee, J. L. (2010). Language, Usage and Cognition. Cambridge University Press. Google Scholar logo with link to Google Scholar
(2011). Usage-based theory and grammaticalization. In H. Narrog & B. Heine (Eds.), The Oxford Handbook of Grammaticalization (pp. 60–78). Oxford University Press. Google Scholar logo with link to Google Scholar
Croft, W. A. (2001). Radical Construction Grammar: Syntactic Theory in Typological Perspective. Oxford University Press. Google Scholar logo with link to Google Scholar
Deng, Y. (2014). Generalized evidence theory. CoRR, abs/1404.4801.v1. Retrieved from [URL]
Denison, D. (2017). Ambiguity and vagueness in historical change. In M. Hundt, S. Molling, & S. E. Pfenniger (Eds.), The Changing English Language: Psycholinguistic Perspectives (pp. 292–318). Cambridge University Press. Google Scholar logo with link to Google Scholar
Diewald, G. (2009). Konstruktionen und Paradigmen [Constructions and paradigms]. Zeitschrift für germanistische Linguistik, 37(3), 445–468. Google Scholar logo with link to Google Scholar
Dipper, S. (2015). Annotierte Korpora für die Historische Syntaxforschung: Anwendungsbeispiele anhand des Referenzkorpus Mittelniederdeutsch [Annotated corpora for historical syntax studies: Applications of the Middle Low German Reference Corpus]. Zeitschrift für Germanistische Linguistik, 43(3), 516–563. Google Scholar logo with link to Google Scholar
Dipper, S., Donhauser, K., Klein, T., Linde, S., Müller, S., & Wegera, K.-P. (2013). HiTS: ein Tagset für historische Sprachstufen des Deutschen [HiTS: A tagset for historical language levels of German]. Journal for Language Technology and Computational Linguistics, 281, 85–137. Google Scholar logo with link to Google Scholar
Dubois, D. (2006). Possibility theory and statistical reasoning. Computational Statistics and Data Analysis, 51(1), 47–69. Google Scholar logo with link to Google Scholar
Dubois, D., & Prade, H. (1988). Possibility Theory. Plenum Press.Google Scholar logo with link to Google Scholar
(1990). Rough fuzzy sets and fuzzy rough sets. International Journal of General Systems, 171, 191–209. Google Scholar logo with link to Google Scholar
Dubois, D., Prade, H., & Smets, P. (1996). Representing partial ignorance. IEEE Transactions on Systems, Man and Cybernetics, Series A, 26(3), 361–377. Google Scholar logo with link to Google Scholar
Eckart de Castilho, R., Mújdricza-Maydt, E., Yimam, S. M., Hartmann, S., Gurevych, I., Frank, A., & Biemann, C. (2016). A web-based tool for the integrated annotation of semantic and syntactic structures. In Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH) (pp. 76–84). The COLING 2016 Organizing Committee. [URL]
Hacking, I. (1975). The Emergence of Probability: A Philosophical Study of Early Ideas about Probability, Induction and Statistical Inference. Cambridge University Press.Google Scholar logo with link to Google Scholar
Hajek, P. (1998). Metamathematics of Fuzzy Logic. Kluwer. Google Scholar logo with link to Google Scholar
Heine, B. (2002). On the role of context in grammaticalization. In I. Wischer & G. Diewald (Eds.), New Reflections on Grammaticalization (pp. 83–101). John Benjamins. Google Scholar logo with link to Google Scholar
Heine, B., & Narrog, H. (2010). Grammaticalization and linguistic analysis. In B. Heine & H. Narrog (Eds.), The Oxford Handbook of Linguistic Analysis (pp. 401–423). Oxford University Press.Google Scholar logo with link to Google Scholar
Keynes, J. M. (1909). A treatise on probability. Diamond, 3(2).Google Scholar logo with link to Google Scholar
Klie, J.-C., Bugert, M., Boullosa, B., de Castilho, R. E., & Gurevych, I. (2018). The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th international conference on computational linguistics: System demonstrations (pp. 5–9). Association for Computational Linguistics. [URL]
Krishnapuram, R. (1994). Generation of membership functions via possibilistic clustering. In Proceedings of the IEEE 3rd International Fuzzy Systems Conference. Google Scholar logo with link to Google Scholar
Kruse, R., Schwecke, E., & Heinsohn, J. (1991). Uncertainty and Vagueness in Knowledge Based Systems. Springer. Google Scholar logo with link to Google Scholar
Kübler, S., & Zinsmeister, H. (2015). Corpus Linguistics and Linguistically Annotated Corpora. Bloomsbury.Google Scholar logo with link to Google Scholar
Lakoff, G. (1987). Cognitive models and prototype theory. In U. Neisser (Ed.), Concepts and Conceptual Development (pp. 63–100). Cambridge University Press.Google Scholar logo with link to Google Scholar
Langacker, R. W. (1987). Foundations of Cognitive Grammar (i): Theoretical Prerequisites. Stanford University Press.Google Scholar logo with link to Google Scholar
Lehmberg, M. (2013). Der Goslarer Ratskodex – das Stadtrecht um 1350 [Codex of Goslar’s Council – the Municipal Law around 1350]. Verlag für Regionalgeschichte.Google Scholar logo with link to Google Scholar
Lientz, B. P. (1972). On time dependent fuzzy sets. Information Sciences, 4(3–4), 367–376. Google Scholar logo with link to Google Scholar
Matheron, G. (1975). Random Sets and Integral Geometry. John Wiley and Sons.Google Scholar logo with link to Google Scholar
Merten, M. (2018). Literater Sprachausbau kognitiv-funktional [Literate language expansion cognitive-functional]. De Gruyter. Google Scholar logo with link to Google Scholar
Merten, M. & Tophinke, D. (2019). Interaktive Analyse historischen Grammatikwandels. Konstruktionsgrammatik trifft auf machine learning [Interactive analysis of historical grammatical change: Construction grammar meets machine learning]. Jahrbuch für Germanistische Sprachgeschichte, 101, 303–323. Google Scholar logo with link to Google Scholar
Narrog, H. (2012). Modality, Subjectivity, and Semantic Change: A Cross-linguistic Perspective. Oxford University Press. Google Scholar logo with link to Google Scholar
Nguyen, H. T. (1978). On random sets and belief functions. Journal of Mathematical Analysis and Applications, 651, 531–542. Google Scholar logo with link to Google Scholar
Pawlak, Z. (1982). Rough sets. International Journal of Computer and Information Sciences, 111, 341–356. Google Scholar logo with link to Google Scholar
Schmid, H.-J. (2010). Does frequency in text instantiate entrenchment in the cognitive system? In D. Glynn & K. Fischer (Eds.), Quantitative Methods in Cognitive Semantics: Corpus-Driven Approaches (pp. 110–133). De Gruyter. Google Scholar logo with link to Google Scholar
Seemann, N., Merten, M., Geierhos, M., Tophinke, D., & Hüllermeier, E. (2017). Annotation challenges for reconstructing the structural elaboration of Middle Low German. In B. Alex, S. Degaetano-Ortlieb, A. Feldman, A. Kazantseva, N. Reiter, & S. Szpakowicz (Eds.), Proceedings of Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, and Literature (pp. 40–45). Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton University Press. Google Scholar logo with link to Google Scholar
Shilkret, N. (1971). Maxitive measure and integration. Nederlandse Akadademie van Wetenschappen. Proceedings Serie A 74 = Indagationes Mathematicae, 331, 109–116. Google Scholar logo with link to Google Scholar
Skala, H. (1978). On many-valued logics, fuzzy sets, fuzzy logics and their applications. Fuzzy Sets and Systems, 1(2), 129–149. Google Scholar logo with link to Google Scholar
Smets, P., & Kennes, R. (1994). The transferable belief model. Artificial Intelligence, 661, 191–234. Google Scholar logo with link to Google Scholar
Taylor, J. R. (2003). Linguistic Categorization. Oxford University Press.Google Scholar logo with link to Google Scholar
Traugott, E., & Trousdale, G. (2010). Gradience, gradualness and grammaticalization: How do they intersect? In E. C. Traugott & G. Trousdale (Eds.), Gradience, Gradualness and Grammaticalization (pp. 19–44). John Benjamins. Google Scholar logo with link to Google Scholar
(2013). Constructionalization and Constructional Changes. Oxford University Press. Google Scholar logo with link to Google Scholar
Trousdale, G. (2012). Grammaticalization, lexicalization and constructionalization from a cognitive-pragmatic perspective. In H.-J. Schmid (Ed.), Cognitive Pragmatics (pp. 533–558). De Gruyter. Google Scholar logo with link to Google Scholar
(2013). Gradualness in language change. In A. G. Ramat, C. Mauri, & P. Molinelli (Eds.), Synchrony and Diachrony: A Dynamic Interface (pp. 27–42). John Benjamins. Google Scholar logo with link to Google Scholar
Walley, P. (1991). Statistical Reasoning with Imprecise Probabilities. Chapman and Hall. Google Scholar logo with link to Google Scholar
Zadeh, L. A. (1965). Fuzzy sets. Information and Control, 8(3), 338–353. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue