Article published In: Functions of Language
Vol. 25:2 (2018) ► pp.229–258
A stepwise method for annotating appraisal
Published online: 19 October 2018
https://doi.org/10.1075/fol.15016.fuo
https://doi.org/10.1075/fol.15016.fuo
Abstract
Despite a growing awareness of methodological issues, the literature on appraisal has not so far provided adequate answers
to some of the key challenges involved in reliably identifying and classifying evaluative language expressions. This article
presents a stepwise method for the manual annotation of appraisal in text that is designed to optimize reliability,
replicability and transparency. The procedure consists of seven steps, from the creation of a context-specific annotation manual
to the statistical analysis of the quantitative data derived from the manually-performed annotations. By presenting this method,
the article pursues the twofold purpose of (i) providing a practical tool that can facilitate more reliable, replicable and
transparent analyses, and (ii) fostering a discussion of the best practices that should be observed when manually annotating
appraisal.
Article outline
- 1.Introduction
- 2.Challenges in annotating appraisal
- 2.1Challenges in identifying appraisal
- 2.2Challenges in classifying appraisal
- 2.3Reliability, replicability, transparency
- 3.The stepwise annotation method
- 3.1General principles
- Principle 1.All choices should be accounted for
- Principle 2.The annotation guidelines should be tested and refined until maximum reliability is achieved
- Principle 3.Reliability should always be assessed, and reliability scores reported and discussed
- 3.2An outline of the steps
- 3.2.1Step 1: Define the scope of the project
- 3.2.2Step 2: Select and configure an annotation tool
- 3.2.3Step 3: Draft a context-specific annotation manual
- 3.2.4Step 4: Assess reliability
- 3.2.5Step 5: Refine the annotation manual
- 3.2.6Step 6: Annotate the corpus
- 3.2.7Step 7: Analyze the results
- 3.1General principles
- 4.Conclusion
- Acknowledgements
- Notes
References
References (55)
Artstein, Ron & Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4). 555–596.
Baayen, R. Harald. 2008. Analyzing linguistic data: A practical introduction to Statistics using R. Cambridge: CUP.
Bartalesi Lenzi, Valentina, Giovanni Moretti & Rachele Sprugnoli. 2012. CAT: the CELCT Annotation Tool. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk & Stelios Piperidis (eds.), Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC12), Istanbul: European Language Resources Association (ELRA). Available online at: [URL] [Accessed 22 September 2015]
Bednarek, Monika. 2006. Evaluation in media discourse: Analysis of a newspaper corpus. London: Continuum.
. 2009. Language patterns and ATTITUDE. Functions of Language 16(2). 165–192.
. 2014. An astonishing season of destiny! Evaluation in blurbs used for advertising TV series. In Geoff Thompson & Laura Alba-Juez (eds.), 197–220.
Ben-Aaron, Diana. 2005. Given and news: Evaluation in newspaper stories about national anniversaries. Text & Talk 25(5). 691–718.
Carretero, Marta & Maite Taboada. 2014. Graduation within the scope of Attitude in English and Spanish consumer reviews of books and movies. In Geoff Thompson & Laura Alba-Juez (eds.), 221–239.
Cohen, Jacob. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1). 37–46.
Cozby, Paul & Scott Bates. 2011. Methods in behavioral research, 11th edn. New York, NY: McGraw-Hill.
Dice, Lee R. 1945. Measures of the amount of ecologic association between species. Ecology 26(3). 297–302.
Divjak, Dagmar. 2006. Ways of intending: Delineating and structuring near synonyms. In Stefan Th. Gries & Anatol Stefanowitsch (eds.), Corpora in cognitive linguistics: Corpus-based approaches to syntax and lexis, 19–56. Berlin: Mouton.
Don, Alexanne. 2007. An approach to the analysis of textual identity through profiles of evaluative disposition. In Proceedings of the Australian Systemic Functional Linguistics Association 2007 Conference. Available at: [URL]. [Accessed 22 September 2015]
Fuoli, Matteo. 2012. Assessing social responsibility: A quantitative analysis of Appraisal in BP’s and IKEA’s social reports. Discourse & Communication 6(1). 55–81.
Fuoli, Matteo & Dylan Glynn. 2013. Computer-assisted manual annotation of evaluative language expressions: Bridging discourse and corpus approaches. Paper presented at the Evaluative Language and Corpus Linguistics Workshop
, Lancaster University, 22 July 2013.
Fuoli, Matteo & Charlotte Hommerberg. 2015. Optimizing transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349.
Fuoli, Matteo & Carita Paradis. 2014. A model of trust-repair discourse. Journal of Pragmatics 741. 52–69.
Geeraerts, Dirk, Stefan Grondelaers & Peter Bakema. 1994. The structure of lexical variation. Berlin: Mouton.
Glynn, Dylan. 2009. Polysemy, syntax, and variation: A usage-based method for Cognitive Semantics. In Vyvyan Evans & Stéphanie Pourcel (eds.), New Directions in Cognitive Linguistics, 77–104. Amsterdam: Benjamins.
Gries, Stefan Th. 1999. Particle movement: A cognitive and functional approach. Cognitive Linguistics 10(2). 105–146.
Hommerberg, Charlotte & Alexanne Don. 2015. Appraisal and the language of wine appreciation: A critical discussion of the potential of the Appraisal framework as a tool to analyse specialised genres. Functions of Language 22(2). 161–191.
Hood, Susan. 2004. Appraising research: Taking a stance in academic writing. Sydney: University of Technology PhD thesis.
Hunston, Susan. 2004. Counting the uncountable: Problems of identifying evaluation in a text and in a corpus. In Alan Partington, John Morley & Louann Haarman (eds.), Corpora and Discourse, 157–188. Bern: Peter Lang.
. 2011. Corpus approaches to evaluation: Phraseology and evaluative language. New York, NY: Taylor & Francis.
Kirk, Jerome & Marc L. Miller. 1986. Reliability and validity in qualitative research. Beverly Hills, CA: Sage.
Krippendorf, Klaus. 2004. Content Analysis: An introduction to its methodology. Thousand Oaks, CA: Sage.
Lipovsky, Caroline. 2008. Constructing affiliation and solidarity in job interviews. Discourse & Communication 2(4). 411–432.
. 2011.
It’s really a great presentation!: Appraising candidates in job interviews. Linguistics and the Human Sciences 4(2). 161–185.
. 2013. Negotiating ones expertise through appraisal in CVs. Linguistics and the Human Sciences 8(3). 307–333.
Mackay, James & Jean Parkinson. 2009.
My very own mission impossible: An APPRAISAL analysis of student teacher reflections on a design and technology project. Text & Talk 29(6). 729–753.
Macken-Horarik, Mary & Anne Isaac. 2014. Appraising Appraisal. In Geoff Thompson & Laura Alba-Juez (eds.), 67–92.
Marshall, Christine, Ralph Adendorff & Vivian de Klerk. 2010. The role of APPRAISAL in the NRF Rating System: An analysis of Judgement and Appreciation in peer reviewers’ reports. Southern African Linguistics and Applied Language Studies 27(4). 391–412.
Martin, J. R. 2000. Beyond exchange: Appraisal systems in English. In Susan Hunston & Geoff Thompson (eds.), Evaluation in text: Authorial stance and the construction of discourse, 142–175. Oxford: OUP.
Martin, J. R. & David Rose. 2003. Working with discourse: Meaning beyond the clause. London: Continuum.
Martin, J. R. & Peter R. R. White. 2005. The language of evaluation: Appraisal in English. London: Palgrave Macmillan.
Mauranen, Anna & Marina Bondi. 2003. Evaluative language use in academic discourse. Journal of English for Academic Purposes 2(4). 269–271.
O’Donnell, Mick. 2008. Demonstration of the UAM CorpusTool for text and image annotation. Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session. 13–16.
. 2014. Exploring identity through Appraisal Analysis: A corpus annotation methodology. Linguistics and the Human Sciences 9(1). 95–116.
Page, Ruth E. 2003. An analysis of APPRAISAL in childbirth narratives with special consideration of gender and storytelling style. Text & Talk 23(2). 211–237.
Paradis, Carita, Joost van de Weijer, Caroline Willners & Magnus Lindgren. 2012. Evaluative polarity of antonyms. Lingue e linguaggio 11(2). 199–214.
Pounds, Gabrina. 2010. Attitude and subjectivity in Italian and British hardnews reporting: The construction of a culture-specific ‘reporter’ voice. Discourse Studies 12(1). 106–137.
. 2011.
This property offers much character and charm: Evaluation in the discourse of online property advertising. Text & Talk 31(2). 195–220.
Pustejovsky, James & Amber Stubbs. 2012. Natural language annotation for machine learning. Sebastopol, CA: O’Reilly Media.
Read, Jonathon & John Carroll. 2012. Annotating expressions of Appraisal in English. Language Resources and Evaluation 46(3). 421–447.
Ryshina-Pankova, Marianna. 2014. Exploring argumentation in course-related blogs through ENGAGEMENT. In Geoff Thompson & Laura Alba-Juez (eds.), 281–302.
Santamaría-García, Carmen. 2014. Evaluative discourse and politeness in university students’ communication through social networking sites. In Geoff Thompson & Laura Alba-Juez (eds.), 387–411.
Taboada, Maite & Marta Carretero. 2012. Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for ATTITUDE applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295.
Taboada, Maite, Marta Carretero & Jennifer Hinnell. 2014. Loving and hating the movies in English, German and Spanish. Languages in Contrast 14(1). 127–161.
Thompson, Geoff. 2014. AFFECT and emotion, target-value mismatches, and Russian dolls: Refining the APPRAISAL model. In Geoff Thompson & Laura Alba-Juez (eds.), 47–66.
Thompson, Geoff & Laura Alba-Juez (eds.). 2014. Evaluation in context. Amsterdam: Benjamins.
Cited by (59)
Cited by 59 other publications
Anisimova, Mariia & Šárka Zikánová
Ben Youssef, Chadi & Antón de la Fuente
Gao, Qingyu, Dezheng (William) Feng & Bekalu Tadesse Moges
Hidalgo‐Downing, Laura
Hoemann, Katie, Yeasle Lee, Èvelyne Dussault, Simon Devylder, Lyle H. Ungar, Dirk Geeraerts & Batja Mesquita
Hsu, Chan-Chia, Yu-Yun Chang & Yun Biao
Li, Long, Shoshana Dreyfus & Alexanne Don
Sikorskii, Sergei & María Luisa Carrió-Pastor
Sun, Shuyi (Amelia) & Feng (Kevin) Jiang
Sun, Shuyi Amelia & Feng (Kevin) Jiang
Allami, Hamid, Monica Karlsson & Hamid Reza Shahroosvand
Cheng, Shi
Del Saz Rubio, María Milagros
Eguchi, Masaki & Kristopher Kyle
Geng, Yifan, Gong Chen & Ming Li
Hidalgo-Downing, Laura & Paula Pérez-Sobrino
Hidalgo-Downing, Laura, Paula Pérez-Sobrino, Laura Filardo-Llamas, Carmen Maíz-Arévalo, Begoña Núñez-Perucha, Alfonso Sánchez-Moya & Julia T. Williams Camus
2024. A protocol for the annotation of evaluative stance and metaphor across four discourse genres. Revista Española de Lingüística Aplicada/Spanish Journal of Applied Linguistics 37:2 ► pp. 486 ff.
Jung, Yejin, Dana Gablasova, Vaclav Brezina & Hanna Schmück
Lu, Rickey
Parviz, Muhammed, Alireza Jalilifar & Alexanne Don
Song, Jiannan
Sun, Shuyi Amelia, Feng Kevin Jiang & Yanhua Liu
Trnavac, Radoslava & Nele Põldvere
Xu, Qingxin
Alwohaibi, Hala Abdulrahman & Hesham Suleiman Alyousef
Carretero, Marta
Cibulskienė, Jurga
2023. Scepticism voiced through extended metaphors. Metaphor and the Social World 13:2 ► pp. 197 ff.
Jing, Yi & Qingxin Xu
Makki, Mohammad & Andrew S. Ross
2023. “We were cocked & loaded to retaliate”. Journal of Language Aggression and Conflict 11:1 ► pp. 1 ff.
Sun, Jing & Zhenqian Liu
Trnavac, Radoslava & Maite Taboada
Watanabe, Hideo
Yu, Wei
Andersson, Marta
2022. ‘So many “virologists” in this thread!’. Pragmatics. Quarterly Publication of the International Pragmatics Association (IPrA) 32:4 ► pp. 489 ff.
Benítez-Castro, Miguel-Ángel & Encarnación Hidalgo-Tenorio
Forceville, Charles & Natalia Sánchez-Querubín
2022. Relevance theory perspectives on web-mediated communication. In Dynamism in Metaphor and Beyond [Metaphor in Language, Cognition, and Communication, 9], ► pp. 325 ff.
FUOLI, MATTEO, JEANNETTE LITTLEMORE & SARAH TURNER
Glynn, Dylan
2022. Emergent categories. In Analogy and Contrast in Language [Human Cognitive Processing, 73], ► pp. 245 ff.
Hansson, Sten, Ruth Page & Matteo Fuoli
Lamb, David A. & Thora Tenbrink
2022. Evaluating Jesus and other “heroes”. Language, Context and Text. The Social Semiotics Forum 4:2 ► pp. 227 ff.
Liu, Yufeng & Dechao Li
Shrestha, Prithvi N.
Simaki, Vasiliki, Eleni Seitanidi & Carita Paradis
Sun, Shuyi & Peter Crosthwaite
Sun, Shuyi Amelia & Peter Crosthwaite
Vilar Lluch, Sara
2022. Redefining attitude for studying explicit and indirect evaluations of human behaviour. Functions of Language 29:2 ► pp. 199 ff.
HIDALGO-TENORIO, ENCARNACIÓN & MIGUEL-ÁNGEL BENÍTEZ-CASTRO
Mavis Ho, Nga-Ki
van Driel, Martine
2021.
Evaluating news events. In Style and Reader Response [Linguistic Approaches to Literature, 36], ► pp. 143 ff.
Kolhatkar, Varada, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla & Maite Taboada
Ross, Andrew S. & David Caldwell
Van Herck, Rebecca, Sofie Decock & Bernard De Clerck
Fusari, Sabrina
Geng, Yifan & Sue Wharton
Page, Ruth
2019. Self-denigration and the mixed messages of ‘ugly’ selfies in Instagram. Internet Pragmatics 2:2 ► pp. 173 ff.
Paternoster, Annick
2019. Politeness and evaluative adjectives in Italian turn-of-the-century etiquette books (1877–1914). In Politeness in Nineteenth-Century Europe [Pragmatics & Beyond New Series, 299], ► pp. 107 ff.
Bisiada, Mario
[no author supplied]
This list is based on CrossRef data as of 9 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
