Cover not available

In:Language and Text: Data, models, information and applications
Edited by Adam Pawłowski, Jan Mačutek, Sheila Embleton and George Mikros
[Current Issues in Linguistic Theory 356] 2021
► pp. 145–162

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

From distinguishability to informativity

A quantitative text model for detecting random texts

 | Goethe University Frankfurt
 | Goethe University Frankfurt
 | Goethe University Frankfurt
 | Goethe University Frankfurt
Published online: 22 December 2021
DOI logo with link
https://doi.org/10.1075/cilt.356.10kon
Abstract
We present a study of the distinctiveness of random and non-random texts based on text characteristics of quantitative linguistics. We additionally experiment with text features that evaluate contiguity associations among sentences by means of BERT (Bidirectional Encoder Representations from Transformers). To this end, we experiment with generative models for random texts as currently discussed in the context of neural networks. The chapter contributes to the clarification of deficits of existing random text models and of the informativeness of quantitative text features.
Article outline
  • 1.Introduction
  • 2.Text corpora and their quantification
    • 2.1Quantification
    • 2.2Text corpora and their randomization
    • 2.3Classification and evaluation methods
  • 3.Results
  • 4.Discussion
  • 5.Conclusion
  • Notes
  • References
References (43)
References
Altmann, Gabriel. 1988. Wiederholungen in Texten. Bochum: Brockmeyer.Google Scholar logo with link to Google Scholar
Baayen, Harald, Hans van Halteren & Fiona Tweedie. 1996. Outside the cave of shadows: Using syntactic annotation to enhance authorship attribution. Literary and Linguistic Computing 11(3). 121–131. Google Scholar logo with link to Google Scholar
Bahdanau, Dzmitry, Kyunghyun Cho & Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.Google Scholar logo with link to Google Scholar
Bengio, Yoshua, Réjean Ducharme, Pascal Vincent & Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research 3. 1137–1155.Google Scholar logo with link to Google Scholar
Biemann, Chris. 2007. A random text model for the generation of statistical language invariants. In Candace Sidner, Tanja Schultz, Matthew Stone & ChengXiang Zhai (eds.), Human language technologies 2007: The conference of the North American chapter of the association for computational linguistics; proceedings of the main conference, 105–112. Rochester, NY: Association for Computational Linguistics.Google Scholar logo with link to Google Scholar
Boser, Bernhard E., Isabelle M. Guyon & Vladimir N. Vapnik. 1992. A training algorithm for optimal margin classifiers. In David Haussler (ed.), Proceedings of the fifth annual workshop on computational learning theory, 144–152. New York: Association for Computing Machinery. Google Scholar logo with link to Google Scholar
Breiman, Leo. 2001. Random forests. Machine Learning 45(1). 5–32. Google Scholar logo with link to Google Scholar
Campolongo, Francesca, Jessica Caribon & Andrea Saltelli. 2007. An effective screening design for sensitivity analysis of large models. Environmental Modelling & Software 22(10). 1509–1518. Google Scholar logo with link to Google Scholar
Čech, Radek. 2015. Text length and the lambda frequency structure of a text. In George K. Mikros & Ján Macutek (eds.), Sequences in language and text, 71–88. Berlin: De Gruyter Mouton. Google Scholar logo with link to Google Scholar
Čech, Radek, Ioan-Iovitz Popescu & Gabriel Altmann. 2013. Methods of analysis of a thematic concentration of the text. Czech and Slovak Linguistic Review 3. 4–21.Google Scholar logo with link to Google Scholar
Chang, Chih-Chung & Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3). 1–27. Google Scholar logo with link to Google Scholar
Cheng, Jianpeng, Li Dong & Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In Jian Su, Kevin Duh & Xavier Carreras (eds.), Proceedings of the 2016 conference on empirical methods in natural language processing, 551–561. Austin, TX: Association for Computational Linguistics. Google Scholar logo with link to Google Scholar
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement 20(1). 37–46. Google Scholar logo with link to Google Scholar
Devlin, Jacob, Ming-Wei Chang, Kenton Lee & Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.Google Scholar logo with link to Google Scholar
Maria Dolores Esteban & Domingo Morales. 1995. A summary on entropy statistics. Kybernetika 31(4). 337–346.Google Scholar logo with link to Google Scholar
Gabrilovich, Evgeniy & Shaul Markovitch. 2006. Overcoming the brittleness bottleneck using Wikipedia: Enhancing text categorization with encyclopedic knowledge. In Proceedings of the twenty-first national conference on artificial intelligence, 2006 Jul 16 (Vol. 6, pp. 1301–1306) Boston, MA: AAAI Press.Google Scholar logo with link to Google Scholar
Hirsch, Jorge E. 2005. An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences 102(46). 16569–16572. Google Scholar logo with link to Google Scholar
Hochreiter, Sepp & Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8). 1735–1780. Google Scholar logo with link to Google Scholar
Joachims, Thorsten. 2002. Learning to classify text using support vector machines. Boston: Kluwer. Google Scholar logo with link to Google Scholar
Kubát, Miroslav, Vladimír Matlach & Radek Čech. 2014. Quita. Quantitative Index Text Analyzer. Lüdenscheid: RAM-Verlag.Google Scholar logo with link to Google Scholar
McIntosh, Robert P. 1967. An index of diversity and the relation of certain concepts to diversity. Ecology 48(3). 392–404. Google Scholar logo with link to Google Scholar
Mehler, Alexander. 2005. Eigenschaften der textuellen Einheiten und Systeme [Properties of textual units and systems]. In Reinhard Köhler, Gabriel Altmann & Rajmund G. Piotrowski (eds.), Quantitative linguistik. ein internationales handbuch / quantitative linguistics. An international handbook, 325–348. Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
Mehler, Alexander, Peter Geibel & Olga Pustylnikov. 2007. Structural classifiers of text types: Towards a novel model of text representation. Journal for Language Technology and Computational Linguistics (JLCL) 22(2). 51–66.Google Scholar logo with link to Google Scholar
Mehler, Alexander, Wahed Hemati, Rüdiger Gleim & Daniel Baumartz. 2018. VienNA: Auf dem Weg zu einer Infrastruktur für die verteilte interaktive evolutionäre Verarbeitung natürlicher Sprache. In Henning Lobin, Roman Schneider & Andreas Witt (eds.), Forschungsinfrastrukturen und digitale Informationssysteme in der germanistischen Sprachwissenschaft, Volume 6, 149–176). Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
Mehler, Alexander, Wahed Hemati, Tolga Uslu & Andy Lücking. 2018. A multidimensional model of syntactic dependency trees for authorship attribution. In Jingyang Jiang & Haitao Liu (eds.), Quantitative analysis of dependency structures, 315–348. Berlin: De Gruyter. Google Scholar logo with link to Google Scholar
Metz, Luke, Ben Poole, David Pfau & Jascha Sohl-Dickstein. 2016. Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163.Google Scholar logo with link to Google Scholar
Morris, Max D. 1991. Factorial sampling plans for preliminary computational experiments. Technometrics 33(2). 161–174. Google Scholar logo with link to Google Scholar
Parzen, Emanuel. 1963. On spectral analysis with missing observations and amplitude modulation. Sankhyā: The Indian Journal of Statistics, Series A, 383–392.Google Scholar logo with link to Google Scholar
Popescu, Ioan-Iovitz. 2009. Word frequency studies, Volume 64. Berlin: Walter de Gruyter.Google Scholar logo with link to Google Scholar
Popescu, Ioan-Iovitz & Gabriel Altmann. 2006. Some aspects of word frequencies. Glottometrics 13. 23–46.Google Scholar logo with link to Google Scholar
. 2007. Writer’s view of text generation. Glottometrics, 15, 71–81.Google Scholar logo with link to Google Scholar
. 2011. Thematic concentration in texts. Issues in quantitative linguistics 2. 110–116.Google Scholar logo with link to Google Scholar
Popescu, Ioan-Iovitz, Radek Čech & Gabriel Altmann. 2011. The lambda-structure of texts. Lüdenscheid: Ram-Verlag.Google Scholar logo with link to Google Scholar
Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei & Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1(8). 9.Google Scholar logo with link to Google Scholar
Reiter, Ehud & Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering 3(1). 57–87. Google Scholar logo with link to Google Scholar
Saltelli, Andrea. 2002. Making best use of model evaluations to compute sensitivity indices. Computer physics communications 145(2). 280–297. Google Scholar logo with link to Google Scholar
Saltelli, Andrea, Paola Annoni, Ivano Azzini, Francesca Campolongo, Marco Ratto & Stefano Tarantola. 2010. Variance based sensitivity analysis of model output. design and estimator for the total sensitivity index. Computer Physics Communications 181(2). 259–270. Google Scholar logo with link to Google Scholar
Smola, Alex J. & Bernhard Schölkopf. 2004. A tutorial on support vector regression. Statistics and computing 14(3). 199–222. Google Scholar logo with link to Google Scholar
Sobol, Ilya M. 2001. Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Mathematics and Computers in Simulation 55(1–3). 271–280. Google Scholar logo with link to Google Scholar
Székely, Gábor J., Maria L. Rizzo & Nail K. Bakirov. 2007. Measuring and testing dependence by correlation of distances. The Annals of Statistics 35(6). 2769–2794. Google Scholar logo with link to Google Scholar
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser & Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008).Google Scholar logo with link to Google Scholar
Wimmer, Gejza. 2005. The type-token-relation. In Reinhard Köhler, Gabriel Altmann & Rajmund G. Piotrowski (eds.), Quantitative Linguistik: Ein internationales Handbuch [Quantitative linguistics: An international handbook], 361–368. Berlin: De Gruyter.Google Scholar logo with link to Google Scholar
Zhu, Yukun, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba & Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, 19–27. Cambridge, MA: IEEE. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue