Cover not available

Article published In: Pragmatics, Humour and the Internet
Edited by Francisco Yus
[Internet Pragmatics 4:1] 2021
► pp. 150–175

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

On the order of processing of humorous tweets with visual and verbal elements

 | University of Burgos
 | Texas A&M University-Commerce
 | Kent State University
 | Texas A&M University-Commerce
Published online: 28 August 2020
DOI logo with link
https://doi.org/10.1075/ip.00060.sim
Abstract
In this paper we examine the order of processing of multimodal tweets (text + image). Using an eye tracker, we collected a sample of 36 participants reading 25 humorous tweets. Our conclusions show that the processing of multimodal humorous tweets is in line with the processing of other multimodal texts. The participants were significantly more likely to start from the image, followed by the caption. Other elements, such as the tweet’s “author” (the user name) or elements outside the tweet’s frame, attracted significantly less and later attention. The participants spent significantly more time gazing at the caption, before moving on to another area. The longer the participants spent looking at the tweet, the less predictable their gaze direction became.
Article outline
  • 1.Introduction
  • 2.Multimodal texts in the eye tracker
  • 3.Attractors
  • 4.Method
    • 4.1Participants
    • 4.2Stimuli
    • 4.3Procedure
  • 5.Results
    • 5.1Descriptive analysis
    • 5.2Inferential analysis
  • 6.Analysis of the results
    • 6.1Summary of multilevel regression modeling
    • 6.2Gaze duration
    • 6.3Conclusion
  • 7.General conclusions
  • Notes
  • References
References (32)
References
Añaños, Elena, and Anna Astals Serés. 2013. “¿Imagen o texto? El poder de captar la atención visual de los elementos gráficos analizado con el Eye tracker [Image or text? The power to capture the visual attention of the graphic elements analyzed with the eye tracker].” Grafica 1(2): 87–98.Google Scholar logo with link to Google Scholar
Attardo, Salvatore. 2020. The Linguistics of Humor: An Introduction. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Attardo, Salvatore, and Jean-Charles Chabanne. 1992. “Jokes as a text type.” HUMOR: International Journal of Humor Research 5(1–2): 165–176. Google Scholar logo with link to Google Scholar
Attardo, Salvatore, Lucy Pickering, Fofo Lomotey, and Shigehito Menjo. 2013. “Multimodality in conversational humor.” Review of Cognitive Linguistics 11(2): 402–416. Google Scholar logo with link to Google Scholar
Attardo, Salvatore, and Victor Raskin. 1991. “Script theory revis(it)ed: Joke similarity and joke representation model.” HUMOR: International Journal of Humor Research 4(3–4): 293–347. Google Scholar logo with link to Google Scholar
Boerman, Sophie C., Edith G. Smit, and Lex van Meurs. 2011. “Attention battle; the abilities of brand, visual, and text characteristics of the ad to draw attention versus the diverting power of the direct magazine context.” In Advances in Advertising Research, vol.2, ed. by Shintaro Okazaki, 295–310. Wiesbaden: Gabler Verlag. Google Scholar logo with link to Google Scholar
Borodo, Michał. 2015. “Multimodality, translation and comics.” Perspectives 23(1): 22–41. Google Scholar logo with link to Google Scholar
Canestrari, Carla. 2018. “The interplay between visual and verbal language in ‘famous last words’ cartoons.” In The Languages of Humor: Verbal, Visual, and Physical Humor, ed. by Arie Sover, 159–166. London: Bloomsbury Academic.Google Scholar logo with link to Google Scholar
Chiaro, Delia. 2005. “Verbally expressed humor and translation: An overview of a neglected field.” HUMOR: International Journal of Humor Research 18(2): 135–145. Google Scholar logo with link to Google Scholar
Foulsham, Tom, Dean Wybrow, and Neil Cohn. 2016. “Reading without words: Eye movements in the comprehension of comic strips.” Applied Cognitive Psychology 30(4): 566–579. Google Scholar logo with link to Google Scholar
Gerhardt, Cornelia. 2009. “Multimodal and intertextual humor in the media reception situation.” In Humor in Interaction, ed. by Neal R. Norrick, and Delia Chiaro, 79–100. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Hamrick, Phillip, and Nick B. Pandža. 2020. “Contributions of semantic and contextual diversity to the word frequency effect in L2 lexical access.” Canadian Journal of Experimental Psychology 74(1): 25–34. Google Scholar logo with link to Google Scholar
Hegarty, Mary. 1992. “Mental animation: Inferring motion from static displays of mechanical systems.” Journal of Experimental Psychology: Learning, Memory, and Cognition 18(5): 1084–1102.Google Scholar logo with link to Google Scholar
Hernández-Méndez, Janet, and Francisco Muñoz-Leiva. 2015. “What type of online advertising is most effective for eTourism 2.0? An eye tracking study based on the characteristics of tourists.” Computers in Human Behavior 501: 618–625. Google Scholar logo with link to Google Scholar
Kaindl, Klaus. 2004. “Multimodality in the translation of humour in comics.” In Perspectives on Multimodality, ed. by Elija Ventola, Cassily Charles, and Martin Kaltenbacher, 173–192. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Kress, Gunther, and Theo Van Leeuwen. 2001. Multimodal Discourse: The Modes and Media of Contemporary Communication. London: Arnold.Google Scholar logo with link to Google Scholar
Li, Qian, Zhouwei Huang, and Kiel Christianson. 2016. “Visual attention toward tourism photographs with text: An eye-tracking study.” Tourism Management 541: 243–258. Google Scholar logo with link to Google Scholar
Liechty, John, Rik Pieters, and Michel Wedel. 2003. “Global and local covert visual attention: Evidence from a Bayesian hidden Markov model.” Psychometrika 68(4): 519–541. Google Scholar logo with link to Google Scholar
Lohse, Gerald L. 1997. “Consumer eye movement patterns on yellow pages advertising.” Journal of Advertising 26(1): 61–73. Google Scholar logo with link to Google Scholar
Pieters, Rik, and Michel Wedel. 2004. “Attention capture and transfer in advertising: Brand, pictorial, and text-size effects.” Journal of Marketing 68(2): 36–50. Google Scholar logo with link to Google Scholar
. 2007. “Goal control of attention to advertising: The Yarbus implication.” Journal of Consumer Research 34(2): 224–233. Google Scholar logo with link to Google Scholar
R: The R Project for Statistical Computing. “The R Foundation.” [URL]
Radach, Ralph, Stefanie Lemmer, Christian Vorstius, Dieter Heller, and Karina Radach. 2003. “Eye movements in the processing of print advertisements.” In The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, ed. by Jukka Hyönä, Ralph Radach, and Heiner Deubel, 609–623. Amsterdam: Elsevier. Google Scholar logo with link to Google Scholar
Raskin, Victor. 1985. Semantics Mechanisms of Humor. Dordrecht: Reidel.Google Scholar logo with link to Google Scholar
Rayner, Keith, Caren M. Rotello, Andrew J. Stewart, Jessica Keir, and Susan A. Duffy. 2001. “Integrating text and pictorial information: Eye movements when looking at print advertisements.” Journal of Experimental Psychology: Applied 7(3): 219–226.Google Scholar logo with link to Google Scholar
Samson, Andrea C., and Oswald Huber. 2007. “The interaction of cartoonist’s gender and formal features of cartoons.” HUMOR: International Journal of Humor Research 20(1): 1–25. Google Scholar logo with link to Google Scholar
Simarro Vázquez, María. 2016. “Mecanismos de humor verbal en Twitter [Mechanisms of verbal humor on Twitter].” Caracteres: estudios culturales y críticos de la esfera digital 5(2): 32–57.Google Scholar logo with link to Google Scholar
Tsakona, Villy. 2009. “Language and image interaction in cartoons: Towards a multimodal theory of humor.” Journal of Pragmatics 411: 1171–1188. Google Scholar logo with link to Google Scholar
Yus, Francisco. 2016. Humour and Relevance. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
. 2019a. “Multimodality in memes: A cyberpragmatic approach.” In Analyzing Digital Discourse. New Insights and Future Directions, ed. by Patricia Bou-Franch, and Pilar Garcés-Conejos Blitvich, 105–131. Cham: Palgrave Macmillan.
. 2019b. Incongruity-Resolution Patterns in Meme Communication. Paper presented at the International Conference on Verbal Humor, University of Alicante, Spain, 23–25 October.
Zanettin, Federico. 2010. “Humour in translated cartoons and comics.” In Translation, Humour and the Media, ed. by Delia Chiaro, 34–52. London: Continuum.Google Scholar logo with link to Google Scholar
Cited by (5)

Cited by five other publications

Geise, Stephanie & Yi Xu
2025. Effects of Visual Framing in Multimodal Media Environments: A Systematic Review of Studies Between 1979 and 2023. Journalism & Mass Communication Quarterly 102:3 ► pp. 796 ff. DOI logo
Yus, Francisco
2023. Meme-Mediated Humorous Communication. In Pragmatics of Internet Humour, ► pp. 245 ff. DOI logo
Yus, Francisco
2023. Internet Humour. In Pragmatics of Internet Humour, ► pp. 59 ff. DOI logo
Yus, Francisco
2025. Shifting from cognitive pragmatics to Internet pragmatics. Revue roumaine de linguistique 2025:1-2 ► pp. 273 ff. DOI logo
[no author supplied]
2022. The Multimodal Performance of Conversational Humor [Figurative Thought and Language, 13], DOI logo

This list is based on CrossRef data as of 12 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue