Cover not available

Article In: Pragmatics: Online-First Articles

A systematic review of reliability in corpus-based metadiscourse studies

ORCiD logo with link | Université de Jendouba | Université de Sfax
ORCiD logo with link | Universidad de Navarra
This content is being prepared for publication; it may be subject to changes.
Abstract
Metadiscourse has been a major focus of research over the last twenty-five years, attracting methodological approaches from the areas of textual pragmatics and discourse studies, many of which are supported by corpus linguistics. A major challenge in corpus-based discourse studies is subjectivity, which may affect their quality and undermine their methodological rigor. To reduce subjectivity and guarantee consistency, assessing coding reliability is essential. This study advocates combining quantitative with qualitative approaches to reliability. We argue that this mixed-method approach will provide a (better) assessment of reliability. To this aim, this methodological synthesis surveyed research covering empirical corpus-based studies on metadiscourse published in indexed and peer-reviewed journals. One major finding is that most studies did not conduct any reliability measure. Issues in reliability accounts were also identified for those that did. Another major finding is a pervasive lack of transparency and comprehensiveness in reliability reports. Recommendations for enhancing reliability are listed.
Article outline
  • 1.Background
    • 1.1Reliability measures: The state of the art
    • 1.2Reliability in corpus-based metadiscourse studies
    • 1.3Methodological synthesis
  • 2.Research objectives
  • 3.Research questions
  • 4.Methodology
    • 4.1Data collection
      • 4.1.1Eligibility criteria
      • 4.1.2Document selection and extraction
    • 4.2Developing the coding protocol
    • 4.3Reliability
    • 4.4Data analysis
  • 5.Results
    • 5.1Overview of reliability measures
    • 5.2Approaches used to assess reliability
      • 5.2.1Type of reliability indices
      • 5.2.2Double coding
      • 5.2.3Scope of reliability
      • 5.2.4Coder information
      • 5.2.5Handling of disagreements
    • 5.3Qualitative assessments of reliability
    • 5.4Issues of reliability
  • 6.Discussion
    • 6.1Absence of ICRMs
    • 6.2Incomplete ICRM reports
    • 6.3Misuse of ICRMs
    • 6.4Scope of reliability
  • 7.Recommendations
  • 8.Limitations and contributions
  • Acknowledgements
  • Notes
  • Author queries
  • References
References (32)
References
Adel, Annelie. 2017. “Remember That Your Reader Cannot Read Your Mind: Problem/Solution-Oriented Metadiscourse in Teacher Feedback on Student Writing.” English for Specific Purposes 45 (January): 54–68. Google Scholar logo with link to Google Scholar
Ahmed, Abdelhamid M., and Xiao Zhang. 2023. “Students’ Voice in L2 English Writing: A Systematic Review of Literature.” Ampersand 101. Google Scholar logo with link to Google Scholar
Belur, Jyoti, Lisa Tompson, Amy Thornton, and Miranda Simon. 2021. “Interrater Reliability in Systematic Review Methodology: Exploring Variation in Coder Decision-Making.” Sociological Methods & Research 50 (2): 837–65. Google Scholar logo with link to Google Scholar
Bouziri, Basma. 2019. “A Corpus-Assisted Genre Analysis of the Tunisian Lecture Corpus: Focus on Metadiscourse.” PhD dissertation. Université Catholique de Louvain.
. 2020a. “Topic Signaling Metadiscourse Devices in the Tunisian Lecture Corpus.” ESP Today 8 (2): 227–49. Google Scholar logo with link to Google Scholar
. 2020b. “A Corpus-Assisted Genre Analysis of the Tunisian Lecture Corpus: An Exploratory Study.” Research in Corpus Linguistics 8 (2): 103–132. Google Scholar logo with link to Google Scholar
. 2021. “A Tripartite Interpersonal Model for Investigating Metadiscourse in Academic Lectures.” Applied Linguistics 42 (5): 970–89. WOS:000727808600008. Google Scholar logo with link to Google Scholar
Callies, Marcus. 2015. “Learner Corpus Methodology.” In The Cambridge Handbook of Learner Corpus Research, edited by Sylviane Granger, Gaetanelle Gilquin, and Fanny Meunier. CUP, 35–55. Google Scholar logo with link to Google Scholar
Chong, Sin Wang, and Luke Plonsky. 2024. “Research Synthesis in Language Education.” In Reference Module in Social Sciences. Elsevier. Google Scholar logo with link to Google Scholar
. 2024. “A Typology of Secondary Research in Applied Linguistics.” Applied Linguistics Review 15 (4): 1569–94. Google Scholar logo with link to Google Scholar
Crismore, Avon, Raija Markkanen, and Margaret S. Steffensen. 1993. “Metadiscourse in Persuasive Writing: A Study of Texts Written by American and Finnish University Students.” Written Communication 10 (1): 39–71. Google Scholar logo with link to Google Scholar
Dey, Ian. 1993. Qualitative Data Analysis: A User-Friendly Guide for Social Scientists. Routledge.Google Scholar logo with link to Google Scholar
Feng, Guangchao Charles. 2015. “Mistakes and How to Avoid Mistakes in Using Intercoder Reliability Indices.” Methodology 11 (1): 13–22. Google Scholar logo with link to Google Scholar
Freelon, Deen. 2010. “ReCal: Intercoder Reliability Calculation as a Web Service.” International Journal of Internet Science 5 (1): 20–33. [URL]
Grisot, Cristina. 2017. “A Quantitative Approach to Conceptual, Procedural and Pragmatic Meaning: Evidence from Inter-Annotator Agreement.” Journal of Pragmatics 117 (August): 245–63. Google Scholar logo with link to Google Scholar
Hober, Nicole, Tülay Dixon, and Tove Larsson. 2023. “Towards Increased Reliability and Transparency in Projects with Manual Linguistic Coding.” Corpora 18 (2): 245–58. Google Scholar logo with link to Google Scholar
Hovy, E., and Julia Lavid. 2010. “Towards a ‘Science’ of Corpus Annotation: A New Methodological Challenge for Corpus Linguistics.” International Journal of Translation 22 (1).Google Scholar logo with link to Google Scholar
Hunston, Susan. 1999. “Evaluation and the Planes of Discourse: Status and Value in Persuasive Texts.” In Evaluation in Text: Authorial Stance and the Construction of Discourse, edited by Geoff Thompson, and Susan Hunston, 176–207. Oxford University Press.Google Scholar logo with link to Google Scholar
Hyland, Ken. 2005. Metadiscourse: Exploring Interaction in Writing. Continuum Discourse Series.Google Scholar logo with link to Google Scholar
Larsson, Tove, Magali Paquot, and Luke Plonsky. 2020. “Inter-Rater Reliability in Learner Corpus Research: Insights from a Collaborative Study on Adverb Placement.” International Journal of Learner Corpus Research 6 (2): 237–51. Google Scholar logo with link to Google Scholar
Liu, Yanhua, and Guangwei Hu. 2021. “Mapping the Field of English for Specific Purposes (1980–2018): A Co-Citation Analysis”. English for Specific Purposes 611: 97–116. Google Scholar logo with link to Google Scholar
Norouzian, Reza. 2021. “Interrater Reliability in Second Language Meta-Analyses: The Case of Categorical Moderators.” Studies in Second Language Acquisition 43 (4): 896–915. Google Scholar logo with link to Google Scholar
Paltridge, Brian, and Aek Phakiti, eds. 2015. Research Methods in Applied Linguistics: A Practical Resource. Bloomsbury Academic.Google Scholar logo with link to Google Scholar
Pearson, William S., and Esmaeel Abdollahzadeh. 2023. “Metadiscourse in Academic Writing: A Systematic Review.” Lingua 2931. Google Scholar logo with link to Google Scholar
Perez Aguilar, Marta, and Elisabeth Macia Arno. 2002. “Metadiscourse in Lecture Comprehension: Does It Really Help Foreign Language Learners?” Atlantis 24 (1): 7–21.Google Scholar logo with link to Google Scholar
Plonsky, Luke, and Frederick Oswald. 2015. “Meta-Analyzing Second Language Research.” In Methodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis. Routledge, 106–128. Google Scholar logo with link to Google Scholar
Plonsky, Luke, and Deirdre J. Derrick. 2016. “A Meta-Analysis of Reliability Coefficients in Second Language Research.” The Modern Language Journal 100 (2): 538–53. Google Scholar logo with link to Google Scholar
Rodwell, Mary K. 1998. Social Work Constructivist Research. Garland Publishing.Google Scholar logo with link to Google Scholar
Schröter, Thorsten, and Agnieszka Jablonska Eklöf. 2020. “Assessment in English for Young Learners in Sweden: Guidelines, Challenges and Coping Strategies.” Educare, no. 3 (October): 1–45. Google Scholar logo with link to Google Scholar
Spooren, Wilbert, and Liesbeth Degand. 2010. “Coding Coherence Relations: Reliability and Validity.” Corpus Linguistics and Linguistic Theory 6 (2). Google Scholar logo with link to Google Scholar
Swales, John M. 2019. “The Futures of EAP Genre Studies: A Personal Viewpoint.” Journal of English for Academic Purposes 38 (March): 75–82. Google Scholar logo with link to Google Scholar
Wood, Margaret, Tove, Larsson, Luke Plonsky, Scott Sterling, Merja Kytö, and Katherine Yaw. 2024. Addressing Questionable Research Practices in Applied Linguistics: A Practical Guide. Applied Linguistics Press.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue