Cover not available

In:Corpus Approaches to Social Media:
Edited by Sofia Rüdiger and Daria Dayter
[Studies in Corpus Linguistics 98] 2020
► pp. 149–174

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
EPUB logo with download linkDownload Book EPUB
authorization required

Chapter 7
Constructing corpora from images and text

An introduction to Visual Constituent Analysis

 | Loughborough University
 | Lancaster University
 | Lancaster University
Published online: 4 November 2020
DOI logo with link
https://doi.org/10.1075/scl.98.07chr
Abstract
Visual analysis represents a significant oversight in the corpus literature, and possibly one that may lead to unintended omissions, particularly when analysing social media. In this chapter we introduce Visual Constituent Analysis (VCA), a method of multimodal corpus construction that allows researchers to construct and analyse visual aspects of online media in large-scale corpora. The chapter addresses the shortcomings of a purely textual approach to discourse analysis when dealing with social media texts and offers a solution using computer ‘Vision’-based image annotation (in our case Google Cloud Vision). Finally, we demonstrate how our approach can be used to analyse a sample of 150,000 micro-blog posts from Twitter and show the difference in level of user interaction with combined image/texts over language-only social media texts.
Article outline
  • 1.Introduction
  • 2.Opportunities and obstacles – Why ‘visual’?
  • 3.Tentative solutions – Constructing ‘constituents’
    • 3.1Vision
    • 3.2Concatenating outputs
  • 4.Analysing T-IRA with VCA
    • 4.1Hostile state information operations
    • 4.2T-IRA – a general overview
  • 5.Quantifying the importance of images
    • 5.1Images, likes and retweets
    • 5.2Text-image overlap
  • 6.T-IRA – A case study
    • 6.1Data
    • 6.2Method
    • 6.3Analysis
      • 6.3.1Image reference (IR)
      • 6.3.2Image and text reference (ITR)
  • 7.Concluding remarks
  • Acknowledgments
  • Notes
  • References
References (38)
References
Allwood, Jens. 2008. Multimodal corpora. In Corpus Linguistics: An International Handbook, Vol. 1 [Handbücher zur Sprach- und Kommunikationswissenschaft 29.1], A. Lüdeling & M. Kytö (eds), 207–225. Berlin: Mouton de Gruyter.Google Scholar logo with link to Google Scholar
Archer, Dawn, Wilson, Andrew & Rayson, Paul. 2002. Introduction to the USAS Category System. Lancaster University. <[URL]> (15 October 2019).
Baker, Paul. 2006. Using Corpora in Discourse Analysis. London: Continuum.Google Scholar logo with link to Google Scholar
Baker, Paul & McEnery, Tony. 2015. Corpora and Discourse Studies: Integrating Discourse and Corpora. London: Palgrave. Google Scholar logo with link to Google Scholar
Bateman, John, Wildfeuer, Jamina & Hiippala, Tuomo. 2017. Multimodality: Foundations, Research and Analysis – A Problem-oriented Introduction. Berlin: De Gruyter. Google Scholar logo with link to Google Scholar
Bednarek, Monika & Caple, Helen. 2017. Introducing a new topology for (multimodal) discourse analysis. In Transforming Contexts: Papers from the 44th International Systemic Functional Congress, Phil Chappell & John S. Knox (eds), 19–25. Wollongong: 44th ISFC Organising Committee.Google Scholar logo with link to Google Scholar
Caple, Helen. 2018. Analysing the multimodal text. In Corpus Approaches to Discourse: A Critical Review, Charlotte Taylor & Anna Marchi (eds), 85–109. London: Routledge. Google Scholar logo with link to Google Scholar
. 2019. “Lucy says today she is a Labordoodle”: How the dogs-of-Instagram reveal voter preferences. Social Semiotics 29(4): 427–447. Google Scholar logo with link to Google Scholar
Chang, Yan-Shou. 2017. Fine-grained attention for image caption generation. Multimedia Tools and Applications 77(7): 2959–2971.Google Scholar logo with link to Google Scholar
Chen, Jianfu, Kuznetsova, Polina, Warren, David S. & Choi, Yejin. 2015. Deja image-captions: A corpus of expressive descriptions in repetition. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Matt Post (ed.), 504–514. Denver CO: ACL.Google Scholar logo with link to Google Scholar
CNSSI 4009–2015. CNSSI 4009 Committee on National Security Systems (CNSS) Glossary. Strategic Environmental Research and Development Program (SERDP), Committee on National Security Systems. <[URL]> (20 October 2019).
Deighton-Smith, Nova & Bell, Beth T. 2018. Objectifying fitness: A content and thematic analysis of #Fitspiration images on social media. Psychology of Popular Media Culture 7(4): 1–41. Google Scholar logo with link to Google Scholar
Facebook. 2016. Facebook reports second quarter 2016 results. 27 July 2016, <[URL]> (20 October 2019).
Fanelli, Gabriele, Gall, Juergen, Romsdorfer, Harald, Weise, Thibaut & van Gool, Luc. 2010. 3D vision technology for capturing multimodal corpora: Chances and challenges. In LREC Workshop on Multimodal Corpora, Rada Mihalcea, Joyce Chai & Anoop Sarkar (eds), 70–73. Valletta: European Language Resources Association (ELRA).Google Scholar logo with link to Google Scholar
Firth, John R. 1957. Papers in Linguistics, 1934–1951. London: OUP.Google Scholar logo with link to Google Scholar
Gatt, Albert, Tanti, Marc, Muscat, Adrian, Paggio, Patrizia, Farrugia, Reuben A., Borg, Claudia, Camilleri, Kenneth, Rosner, Micahel & van der Plas, Lonneke. 2018. Face2Text: Collecting an annotated image description corpus for the generation of rich face descriptions. In Proceedings of the 11th edition of the Language Resources and Evaluation Conference, Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, Takenobu Tokunaga (eds), 3323–3328. Miazaki: European Language Resources Association (ELRA).Google Scholar logo with link to Google Scholar
Jewitt, Carey. 2015. Multimodal analysis. In The Routledge Handbook of Language and Digital Communication, Alexandra Georgakopoulou & Tereza Spilioti (eds), 69–85. London: Routledge.Google Scholar logo with link to Google Scholar
Knight, Dawn. 2015. e-Language: Communication in the digital age. In Corpora and Discourse Studies, Paul Baker & Tony McEnery (eds), 20–40. London: Palgrave Macmillan. Google Scholar logo with link to Google Scholar
Kress, Gunther & van Leeuwen, Theo. 2006. Reading Images: The Grammar of Visual Design, 2nd edn. London: Routledge. Google Scholar logo with link to Google Scholar
Kuznetsova, Polina, Ordonez, Vicente, Berg, Alexander, Berg, Tamara & Choi, Yejin. 2013. Generalizing image captions for image-text parallel corpus. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Hinrich Schuetze, Pascale Fung & Massimo Poesio (eds), 790–796. Sofia: Association for Computational Linguists (ACL).Google Scholar logo with link to Google Scholar
Liu, Jeffrey, Weinert, Andrew & Amin, Saurabh. 2018. Semantic topic analysis of traffic camera images. In 21st IEEE International Conference on Intelligent Transportation Systems (ITSC), Wei-Bin Zhang, Alexandre M. Bayen, Javier J. Sánchez Medina & Matthew J. Barth (eds), 568–574. Maui: Institute of Electical and Electronics Engineers (IEEE).Google Scholar logo with link to Google Scholar
McGowan, Michael. 2018. ‘Just not blond’: Fake Black Lives Matter Facebook page run by Australian union official – Report, The Guardian. 10 April 2018, <[URL]> (28 November 2019).
Mitchell, William J. T. 1994. Picture Theory: Essays on Verbal and Visual Representation. Chicago IL: University of Chicago Press.Google Scholar logo with link to Google Scholar
Ordonez, Vicente, Kulkarni, Girish & Berg, Tamara L. 2011. Im2Text: Describing images using 1 million captioned photographs. In Advances in Neural Information Processing Systems, John Shawe-Taylor, Richard Zemel, Peter Bartlett, Fernando Pereira & Kilian Weinberger (eds), 1143–1151. Granada: Neural Information Processing Systems (NIPS).Google Scholar logo with link to Google Scholar
Pastra, Katerina & Wilks, Yorick. 2004. Vision-language integration in AI: A reality check. In Proceedings of the 16th European Conference in Artifical Intelligence, Ramón López de Mántaras & Lorenza Saitta (eds), 937–941. Valencia: IOS Press.Google Scholar logo with link to Google Scholar
Pew Research Center. 2019. State of the Union 2019: How Americans see major national issues. 4 February 2019, <[URL]> (28 October 2019).
Rayson, Paul. 2009. Wmatrix: A web-based corpus processing environment. Computing Department, Lancaster University. <[URL]> (5 November 2019).
Rayson, Paul, Archer, Dawn, Piao, Scott & McEnery, Tony. 2004. The UCREL semantic analysis system. In Proceedings of the LREC-04 Workshop, beyond Named Entity Recognition Semantic Labelling for NLP Tasks, Lisbon, Portugal, Maria Teresa Lino, Maria Francisca Xavier, Fátima Ferreira, Rute Costa & Raquel Silva (eds), 7–12. Lisbon: European Language Resource Association (ELRA).Google Scholar logo with link to Google Scholar
Seo, Hyunjin. 2014. Visual propaganda in the age of social media: An empirical analysis of Twitter images during the 2012 Israeli–Hamas conflict. Visual Communication Quarterly 21(3): 150–161. Google Scholar logo with link to Google Scholar
Twitter. 2015. Hearts on Twitter. 3 November 2015, <[URL]> (3 February 2020).
. 2018. An update on our elections integrity work. 1 October 2018, <[URL]> (5 October 2019).
. 2019a. What is a retweet? Twitter Help Center. <[URL]> (3 February 2020).
. 2019b. How to like a tweet. Twitter Help Center. <[URL]> (3 February 2020).
. 2020. Pricing: API access that scales with you and your solution. <[URL]> (10 February 2020).
United States Department of Justice. 2018. Case 1:18-cr-00032-DLF: UNITED STATES OF AMERICA v. INTERNET RESEARCH AGENCY LLC. 16 January 2018, <[URL]> (10 December 2019).
United States Joint Chiefs of Staff. 2014. Information operations. Homeland Security Digital Library. <[URL]> (10 December 2019).
Zappavigna, Michele & Martin, James R. 2018. #Communing affiliation: Social tagging as a resource for aligning around values in social media. Discourse, Context & Media 22: 4–12. Google Scholar logo with link to Google Scholar
Zappavigna, Michele. 2012. Discourse of Twitter and Social Media. London: Continuum.Google Scholar logo with link to Google Scholar
Cited by (7)

Cited by seven other publications

Baker, Paul, Hanna Schmück & Yufang Qian
2025. Automatic Image Tagging for Corpus Linguistics, DOI logo
Christiansen, Alex
2025. Book review: Empirical Multimodality Research: Methods, Evaluations, Implications PflaegingJanaWildfeuerJaninaBatemanJohn, Empirical Multimodality Research: Methods, Evaluations, Implications. Berlin: De Gruyter, 2021. ISBN 978-3-11-072491-2; e-ISBN (EPUB) 978-3-11-072515-5. Visual Communication 24:2 ► pp. 527 ff. DOI logo
Collins, Luke C & Paul Baker
2025. A computer-assisted analysis of image representations of obesity: comparing UK news content with the World Obesity Federation Image Bank. Visual Communication 24:4 ► pp. 889 ff. DOI logo
Dance, William
2025. Corpus Linguistics and Social Media. In Reference Module in Social Sciences, DOI logo
Jadhav, Vaijayanta & Sanjay Krishnapratap Pawar
2025. Computer vision-based analysis of images in university-curated Instagram posts: a user-engagement perspective. Cogent Education 12:1 DOI logo
Sha, Yuze & Beth Malory
2025. Using ATLAS.ti for constructing and analysing multimodal social media corpora. Linguistics Vanguard DOI logo
Hiippala, Tuomo
2024. Rethinking multimodal corpora from the perspective of Peircean semiotics. Frontiers in Communication 9 DOI logo

This list is based on CrossRef data as of 1 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue