Cover not available

Article published In: Target
Vol. 37:4 (2025) ► pp.590–626

Get fulltext from our e-platform
JBE logo with link
EPUB logo with download linkDownload EPUB
authorization required

Modeling rater cognition in translation assessment

An exploratory investigation based on think-aloud, eye-tracking, and interview data

ORCiD logo with link | National University of Singapore | Xiamen University
ORCiD logo with link | Xiamen University
ORCiD logo with link | Renmin University of China
Published online: 11 September 2025
DOI logo with link
https://doi.org/10.1075/target.23040.han
Abstract
In this exploratory study, we investigated rater cognition in English–Chinese translation assessment, drawing on think-aloud, eye-tracking, and interview data. We designed a 3 × 2 × 2 experiment in which experienced raters assessed eighteen renditions of three levels of quality for each translation direction, using a Likert-type scale or analytic rubric scale. We found that: (a) the raters heeded meaning transfer more frequently than other contents; (b) they utilized a variety of processing actions, but a core subset involving eight actions constituted the mainstay; (c) to make a scoring decision, the raters mainly consulted the source text, the target texts, and the rating scale, but also displayed other patterns of interaction (e.g., relying on target texts only); (d) they fixated more frequently per time unit and proportionally longer on the target texts; and (e) translation direction and scoring method seemed to have modulated rater cognition. The implications of these findings for translation assessment are discussed.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Raters’ scoring process in writing assessment
    • 2.2Translation assessment: The role of scoring methods and translation directionality
    • 2.3Rater cognition in translation assessment
  • 3.Method
    • 3.1The rater cognition project and the current study
    • 3.2Participants
    • 3.3Materials
    • 3.4Experimental design
    • 3.5Procedures
    • 3.6Data analysis
  • 4.Results
    • 4.1What aspects of the TTs do raters heed?
    • 4.2What specific processing actions do raters utilize?
    • 4.3What are raters’ overall patterns of interaction with the scoring materials?
    • 4.4How do raters gaze at the three scoring-related materials?
  • 5.Discussion
    • 5.1Discussion of research results
    • 5.2Pedagogical and practical implications
  • 6.Conclusion
  • Notes
  • References
References (47)
References
Angelelli, Claudia V. 2009. “Using a Rubric to Assess Translation Ability: Defining the Construct.” In Testing and Assessment in Translation and Interpreting Studies, edited by Claudia V. Angelelli and Holly E. Jacobson, 13–47. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Baker, Beverly Anne. 2012. “Individual Differences in Rater Decision-Making Style: An Exploratory Mixed-Methods Study.” Language Assessment Quarterly 9 (3): 225–248. Google Scholar logo with link to Google Scholar
Barkaoui, Khaled. 2007. “Rating Scale Impact on EFL Essay Marking: A Mixed-Method Study.” Assessing Writing 12 (2): 86–107. Google Scholar logo with link to Google Scholar
. 2010. “Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience.” Language Assessment Quarterly 7 (1): 54–74. Google Scholar logo with link to Google Scholar
Chen, Jing, Huabo Yang, and Chao Han. 2022. “Holistic Versus Analytic Scoring of Spoken-Language Interpreting: A Multi-Perspectival Comparative Analysis.” The Interpreter and Translator Trainer 16 (4): 558–576. Google Scholar logo with link to Google Scholar
Chesterman, Andrew. 1998. “Causes, Translations, Effects.” Target 10 (2): 201–230. Google Scholar logo with link to Google Scholar
Crisp, Victoria. 2010. “Towards a Model of the Judgement Processes Involved in Examination Marking.” Oxford Review of Education 36 (1): 1–21. Google Scholar logo with link to Google Scholar
Cumming, Alister. 1990. “Expertise in Evaluating Second Language Compositions.” Language Testing 7 (1): 31–51. Google Scholar logo with link to Google Scholar
Cumming, Alister, Robert Kantor, and Donald E. Powers. 2002. “Decision Making While Rating ESL/EFL Writing Tasks: A Descriptive Framework.” The Modern Language Journal 86 (1): 67–96. Google Scholar logo with link to Google Scholar
DeRemer, Mary L. 1998. “Writing Assessment: Raters’ Elaboration of the Rating Task.” Assessing Writing 5 (1): 7–29. Google Scholar logo with link to Google Scholar
Eyckmans, June, and Philippe Anckaert. 2017. “Item-Based Assessment of Translation Competence: Chimera of Objectivity versus Prospect of Reliable Measurement.” Linguistica Antverpiensia, New Series: Themes in Translation Studies 161: 40–56.Google Scholar logo with link to Google Scholar
Feng, Jia. 2018. 中英双向互译中翻译认知过程研究: 基于眼动追踪和键盘记录的实证分析 [Cognitive processing in bidirectional Chinese-English translation: Empirical evidence from eye-tracking and keystroke logging]. Beijing: Foreign Language Teaching and Research Press.Google Scholar logo with link to Google Scholar
Gieshoff, Anne Catherine, and Michaela Albl-Mikasa. 2022. “Interpreting Accuracy Revisited: A Refined Approach to Interpreting Performance Analysis.” Perspectives 32 (2): 210–228. Google Scholar logo with link to Google Scholar
Godfroid, Aline. 2020. Eye Tracking in Second Language Acquisition and Bilinguialism: A Research Synthesis and Methodological Guide. New York: Routledge.Google Scholar logo with link to Google Scholar
Grbić, Nadja. 2008. “Constructing Interpreting Quality.” Interpreting 10 (2): 232–257. Google Scholar logo with link to Google Scholar
Han, Chao, and Xiao Zhao. 2021. “Accuracy of Peer Ratings on the Quality of Spoken-Language Interpreting.” Assessment and Evaluation in Higher Education 46 (8): 1299–1313. Google Scholar logo with link to Google Scholar
Han, Chao, Binghan Zheng, Mingqing Xie, and Shirong Chen. 2024. “Raters’ Scoring Process in Assessment of Interpreting: An Empirical Study Based on Eye Tracking and Retrospective Verbalization.” Interpreter and Translator Trainer 18 (3): 400–422. Google Scholar logo with link to Google Scholar
Han, Chao. 2020. “Translation Quality Assessment: A Critical Methodological Review.” The Translator 26 (3): 257–273. Google Scholar logo with link to Google Scholar
Holmqvist, Kenneth, Saga Lee Örbom, Ignace T. C. Hooge, Diederick C. Niehorster, Robert G. Alexander, Richard Andersson, Jeroen S. Benjamins, et al. 2023. “Eye Tracking: Empirical Foundations for a Minimal Reporting Guideline.” Behavior Research Methods 551: 364–416. Google Scholar logo with link to Google Scholar
House, Juliane. 1997. Translation Quality Assessment: A Model Revisited. Tübingen: Gunter Narr Verlag.Google Scholar logo with link to Google Scholar
Huertas Barros, Elsa, and Juliet Vine. 2018. “Current Trends on MA Translation Courses in the UK: Changing Assessment Practices on Core Translation Modules.” Interpreter and Translator Trainer 12 (1): 5–24. Google Scholar logo with link to Google Scholar
Jakobsen, Arnt Lykke. 2017. “Translation Process Research.” The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 19–49. Hoboken: Wiley-Blackwell. Google Scholar logo with link to Google Scholar
Koby, Geoffrey S. 2015. “The ATA Flowchart and Framework as a Differentiated Error-Marking Scale in Translation Teaching.” In Handbook of Research on Teaching Methods in Language Translation and Interpretation, edited by Ying Cui and Wei Zhao, 220–253. Hershey: IGI Global. Google Scholar logo with link to Google Scholar
Kruger, Haidee, and Jan-Louis Kruger. 2017. “Cognition and Reception.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 71–89. Hoboken: Wiley-Blackwell. Google Scholar logo with link to Google Scholar
Lai, Tzu-Yun. 2011. “Reliability and Validity of a Scale-based Assessment for Translation Tests.” Meta 56 (3): 713–722. Google Scholar logo with link to Google Scholar
Li, Hang, and Lianzhen He. 2015. “A Comparison of EFL Raters’ Essay-Rating Processes across Two Types of Rating Scales.” Language Assessment Quarterly 12 (2): 178–212. Google Scholar logo with link to Google Scholar
Lumley, Tom. 2002. “Assessment Criteria in a Large-Scale Writing Test: What Do They Really Mean to the Raters?” Language Testing 19 (3): 246–276. Google Scholar logo with link to Google Scholar
Ma, Xingcheng, and Dechao Li. 2020. “翻译教师和普通读者在译文在线评阅中的认知过程研究:基于眼动追踪数据的翻译质量评测 [Cognitive processes of translation teachers and ordinary readers in reading translated texts: An eye-tracking perspective to translation quality assessment]” Foreign Languages Research 41: 28–36.Google Scholar logo with link to Google Scholar
Muñoz, Ricardo. 2010. “Leave No Stone Unturned: On the Development of Cognitive Translatology.” Translation and Interpreting Studies 5 (2): 145–162. Google Scholar logo with link to Google Scholar
Muñoz, Ricardo, and Tomás Conde. 2007. “Effects of Serial Translation Evaluation.” In Translationsqualität [Translation quality], edited by Peter A. Schmit and Heike E. Jüngst, 428–444. Frankfurt: Peter Lang.Google Scholar logo with link to Google Scholar
Obdržálková, Vanda. 2018. “Directionality in Translation: Qualitative Aspects of Translation from and into English as a Non-Mother Tongue.” Sendebar 291: 35–57. Google Scholar logo with link to Google Scholar
Pöchhacker, Franz. 2002. “Researching Interpreting Quality: Models and Methods.” In Interpreting in the 21st Century: Challenges and Opportunities, edited by Giuliana Garzone and Maurizio Viezzi, 95–106. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Pokorn, Nike K., Jason Blake, Donald Reindl, and Agnes Pisanski Peterlin. 2020. “The Influence of Directionality on the Quality of Translation Output in Educational Settings.” The Interpreter and Translator Trainer 14 (1): 58–78. Google Scholar logo with link to Google Scholar
Rothe-Neves, Rui. 2008. “Translation Quality Assessment for Research Purposes: An Empirical Approach.” Cadernos de Tradução [Translation notebooks] 2 (10): 113–131.Google Scholar logo with link to Google Scholar
Sun, Sanjun, and Gregory M. Shreve. 2014. “Measuring Translation Difficulty: An Empirical Study.” Target 26 (1): 98–127. Google Scholar logo with link to Google Scholar
Turner, Barry, Miranda Lai, and Neng Huang. 2010. “Error Deduction and Descriptors: A Comparison of Two Methods of Translation Test Assessment.” Translation & Interpreting 2 (1): 11–23.Google Scholar logo with link to Google Scholar
Waddington, Christopher. 2001. “Should Translations Be Assessed Holistically or through Error Analysis?” Hermes: Journal of Linguistics 14 (26): 15–38.Google Scholar logo with link to Google Scholar
Walker, Callum. 2019. “A Cognitive Perspective on Equivalent Effect: Using Eye Tracking to Measure Equivalence in Source Text and Target Text Cognitive Effects on Readers.” Perspectives 27 (1): 124–143. Google Scholar logo with link to Google Scholar
Whyatt, Bogusława. 2019. “In Search of Directionality Effects in the Translation Process and in the End Product.” Translation, Cognition and Behavior 2 (1): 79–100.Google Scholar logo with link to Google Scholar
Winke, Paula, and Hyojung Lim. 2015. “ESL Essay Raters’ Cognitive Processes in Applying the Jacobs et al. Rubric: An Eye-Movement Study.” Assessing Writing 251: 37–53. Google Scholar logo with link to Google Scholar
Wolfe, Edward W. 1997. “The Relationship between Essay Reading Style and Scoring Proficiency in a Psychometric Scoring System.” Assessing Writing 4 (1): 83–106. Google Scholar logo with link to Google Scholar
2005. “Uncovering Rater’s Cognitive Processing and Focus Using Think-Aloud Protocols.” Journal of Writing Assessment 2 (1): 37–56.Google Scholar logo with link to Google Scholar
Zhang, Jie. 2016. “Same Text Different Processing? Exploring How Raters’ Cognitive and Meta-Cognitive Strategies Influence Rating Accuracy in Essay Scoring.” Assessing Writing 271: 37–53. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue