Cover not available

Article published In: Interpreting
Vol. 21:2 (2019) ► pp.245–269

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Holistic assessment of consecutive interpretation

How interpreter trainers rate student performances

 | Hankuk University of Foreign Studies
Published online: 11 November 2019
DOI logo with link
https://doi.org/10.1075/intp.00029.lee
Abstract
The article aims to show how interpreter trainers holistically grade student performances. For this purpose, experimental rating sessions were held for four undergraduate interpreter trainers. The raters were asked to think aloud their quality judgments while holistically assessing six recordings of consecutive interpretation. Their concurrent verbal reports, along with reflective reports, interview transcripts, and video recordings of computer screen activity, were collected and analysed in detail. Findings revealed various facets of interpreting performance assessment, including what procedures the raters followed, what aspects of the performance they focused on, what criteria they depended on for their judgment decisions, and why two ratings of the same performance were divergent. This article also presents a tentative model for holistic rating of consecutive interpretation.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Procedural dimension of interpreting performance assessment
    • 2.2Thinking aloud as a data elicitation method
  • 3.Method
    • 3.1Participants
    • 3.2Task materials
    • 3.3Procedures
    • 3.4Coding schemes
    • 3.5Data analysis
  • 4.Results
    • 4.1Lengths of the verbal protocols
    • 4.2Grades
    • 4.3Rating styles and methods
      • 4.3.1R1
      • 4.3.2R2
      • 4.3.3R3
      • 4.3.4R4
    • 4.4Distribution of positive and negative comments by protocol code
      • 4.4.1R1
      • 4.4.2R2
      • 4.4.3R3
      • 4.4.4R4
    • 4.5A tentative model of the holistic rating process
  • 5.Concluding remarks
  • Acknowledgements
  • Notes
  • References
References (57)
References
Bartłomiejczyk, M. (2007). Interpreting quality as perceived by trainee interpreters: Self-evaluation. The Interpreter and Translator Trainer 1 (2), 247–267. Google Scholar logo with link to Google Scholar
Bejar, I. I. (2012). Rater cognition: Implications for validity. Educational Measurement: Issues and Practice 31 (3), 2–9. Google Scholar logo with link to Google Scholar
Bowles, M. A. (2010). The think-aloud controversy in second language research. London: Routledge. Google Scholar logo with link to Google Scholar
Bühler, H. (1986). Linguistic (semantic) and extralinguistic (pragmatic) criteria for the evaluation of conference interpretation and interpreters. Multilingua 5 (4), 231–235.Google Scholar logo with link to Google Scholar
Chabasse, C. & Kader, S. (2014). Putting interpreting admissions exams to the test. Interpreting 16 (1), 19–33. Google Scholar logo with link to Google Scholar
Chiaro, D. & Nocella, G. (2004). Interpreters’ perception of linguistic and non-linguistic factors affecting quality: A survey through the World Wide Web. Meta 49 (2), 279–293. Google Scholar logo with link to Google Scholar
Choi, J. Y. (2013). Assessing the impact of text length on consecutive interpreting. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 85–96.Google Scholar logo with link to Google Scholar
Coffey, A. & Atkinson, P. (1996). Making sense of qualitative data: Complementary research strategies. Thousand Oaks, CA: Sage.Google Scholar logo with link to Google Scholar
Cohen, A. D. (2000). Exploring strategies in test-taking: Fine-tuning verbal reports from respondents. In G. Ekbatani & H. Pierson (Eds.), Learner-directed assessment in ESL. Mahwah, NJ: Lawrence Erlbaum, 127–150.Google Scholar logo with link to Google Scholar
De Gregoris, G. (2014). The limits of expectations vs. assessment questionnaire-based surveys on simultaneous interpreting quality: The need for a gestaltic model of perception. Rivista internazionale di tecnica della traduzione 161, 57–87.Google Scholar logo with link to Google Scholar
DeRember, M. L. (1998). Writing assessment: Raters’ elaboration of the rating task. Assessing Writing 5 (1), 7–29. Google Scholar logo with link to Google Scholar
Englund Dimitrova, B. & Tiselius, E. (2014). Retrospection in interpreting and translation: Explaining the process? MonTI Special Issue 11, 177–200. Google Scholar logo with link to Google Scholar
Ericsson, K. A. & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). Cambridge, MA: MIT Press. Google Scholar logo with link to Google Scholar
(1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture and Activity 5 (3), 178–186. Google Scholar logo with link to Google Scholar
Eyckmans, J., Anckaert, P. & Segers, W. (2016). Translation and interpretation skills. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment. Berlin: Walter de Gruyter, 219–235. Google Scholar logo with link to Google Scholar
Garzone, G. (2003). Reliability of quality criteria evaluation in survey research. In A. Collados Aís, M. M. Fernández Sánchez & D. Gile (Eds.), La evaluación de la calidad en interpretación: Investigación. Granada: Comares.Google Scholar logo with link to Google Scholar
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
Iglesias Fernández, E. (2013). Unpacking delivery criteria in interpreting quality assessment. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 51–66.Google Scholar logo with link to Google Scholar
Ivanova, A. (2000). The use of retrospection in research on simultaneous interpreting. In S. Tirkkonen-Condit & R. Jääskeläinen (Eds.), Tapping and mapping the processes of translation and interpreting: Outlooks on empirical research. Amsterdam: John Benjamins, 27–52. Google Scholar logo with link to Google Scholar
Jakobsen, A. L. (2003). Effects of think aloud on translation speed, revision, and segmentation. In F. Alves (Ed.), Triangulating translation: Perspectives in process-oriented research. Amsterdam: John Benjamins, 69–95. Google Scholar logo with link to Google Scholar
Jarvella, R. J., Jensen, A., Jensen, E. H. & Anderson, M. S. (2002). Towards characterizing translator expertise, knowledge and know-how: Some findings using TAPs and experimental methods. In A. Riccardi (Ed.), Translation studies: Perspectives on an emerging discipline. Cambridge: Cambridge University Press, 172–197.Google Scholar logo with link to Google Scholar
Jourdenais, R. (2001). Cognition, instruction and protocol analysis. In P. Robinson (Ed.), Cognition and second language instruction. Cambridge: Cambridge University Press, 354–375. Google Scholar logo with link to Google Scholar
Künzli, A. (2007). Translation revision: A study of the performance of ten professional translators revising a legal text. In Y. Gambier, M. Shlesinger & R. Stolze (Eds.), Doubts and directions in translation studies. Amsterdam: John Benjamins, 115–126. Google Scholar logo with link to Google Scholar
Lee, J. (2009). Toward more reliable assessment of interpreting performance. In S. Hale, U. Ozolins & L. Stern (Eds.), The Critical Link 5: Quality in interpreting – a shared responsibility. Amsterdam: John Benjamins, 171–185. Google Scholar logo with link to Google Scholar
Lee, S.-B. (2014). An interpreting self-efficacy (ISE) scale for undergraduate students majoring in consecutive interpreting: Construction and preliminary validation. The Interpreter and Translator Trainer 8 (2), 183–203. Google Scholar logo with link to Google Scholar
Li, D. (2004). Trustworthiness of think-aloud protocols in the study of translation processes. International Journal of Applied Linguistics 14 (3), 301–313. Google Scholar logo with link to Google Scholar
Liu, M. (2013). Design and analysis of Taiwan’s interpretation certification examination. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 163–178.Google Scholar logo with link to Google Scholar
(2015). Assessment. In F. Pöchhacker (Ed.), Routledge encyclopedia of interpreting studies. London: Routledge, 20–22.Google Scholar logo with link to Google Scholar
Lörscher, W. (2005). The translation process: Methods and problems of its investigation. Meta 50 (2), 597–608. Google Scholar logo with link to Google Scholar
Miles, M. B., Huberman, A. M. & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar logo with link to Google Scholar
Naumenko, O. (2015). Improving performance assessment score validation practices: An instructional module on generalizability theory. Working Papers on Language and Diversity in Education I (1), 1–17.Google Scholar logo with link to Google Scholar
O’Hagan, S. (2014). Variability in assessor responses to undergraduate essays: An issue for assessment quality in higher education. Bern: Peter Lang.Google Scholar logo with link to Google Scholar
Pöchhacker, F. (2001). Quality assessment in conference and community interpreting. Meta 46 (2), 410–425. Google Scholar logo with link to Google Scholar
(2016). Introducing interpreting studies (2nd ed.). London: Routledge. Google Scholar logo with link to Google Scholar
Pressley, M. & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar logo with link to Google Scholar
Rallis, S. F. & Rossman, G. B. (2003). Mixed methods in evaluation contexts: A pragmatic framework. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research. Thousand Oaks, CA: Sage, 491–512.Google Scholar logo with link to Google Scholar
Russell, D. & Winston, B. (2014). Tapping into the interpreting process: Using participant reports to inform the interpreting process in educational settings. Translation & Interpreting 6 (1), 102–127.Google Scholar logo with link to Google Scholar
Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar logo with link to Google Scholar
Saldanha, G. & O’Brien, S. (2013). Research methodologies in translation studies. London: Routledge.Google Scholar logo with link to Google Scholar
Sun, S. (2011). Think-aloud-based translation process research: Some methodological considerations. Meta 56 (4), 928–951. Google Scholar logo with link to Google Scholar
Tiselius, E. (2009). Revisiting Carroll’s scales. In C. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Van Someren, M. W., Barnard, Y. F. & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to modeling cognitive processes. London: Academic Press.Google Scholar logo with link to Google Scholar
Vik-Tuovinen, G. V. (2002). Retrospection as a method of studying the process of simultaneous interpreting. In G. Garzone & M. Viezzi (Eds.), Interpreting in the 21st century: Challenges and opportunities. Amsterdam: John Benjamins, 63–72. Google Scholar logo with link to Google Scholar
Wallace, M. (2013). Rethinking bifurcated testing models in the court interpreter certification process. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 67–83.Google Scholar logo with link to Google Scholar
Wang, J., Napier, J., Goswell, D. & Carmichael, A. (2015). The design and application of rubrics to assess signed language interpreting performance. The Interpreter and Translator Trainer 9 (1), 83–103. Google Scholar logo with link to Google Scholar
Wigglesworth, G. (2005). Current approaches to researching second language learner processes. Annual Review of Applied Linguistics 251, 90–111. Google Scholar logo with link to Google Scholar
Wu, F. S. (2013). How do we assess students in the interpreting examinations? In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt: Peter Lang, 15–33.Google Scholar logo with link to Google Scholar
Zwischenberger, C. (2010). Quality criteria in simultaneous interpreting: An international vs. a national view. The Interpreters’ Newsletter 151, 127–142.Google Scholar logo with link to Google Scholar
Cited by (13)

Cited by 13 other publications

Chen, Shirong & Chao Han
2025. The Role of Annotating in Peer Assessment of Interpreting: A Quasi-experimental Investigation. The Asia-Pacific Education Researcher 34:6 ► pp. 2109 ff. DOI logo
Jiang, Zhaokun & Ziyin Zhang
2025. From black box to transparency: Enhancing automated interpreting assessment with explainable AI in college classrooms. Research Methods in Applied Linguistics 4:3 ► pp. 100237 ff. DOI logo
Achkasov, Andrei Valentinovich
2024. Speech characteristics of consecutive interpreting involving speech recognition tools. Philology. Theory & Practice 17:9 ► pp. 3355 ff. DOI logo
Han, Chao, Binghan Zheng, Mingqing Xie & Shirong Chen
2024. Raters’ scoring process in assessment of interpreting: an empirical study based on eye tracking and retrospective verbalisation. The Interpreter and Translator Trainer 18:3 ► pp. 400 ff. DOI logo
Moratto, Riccardo & Zhimiao Yang
2024. Probing the cognitive load of consecutive interpreters. Translation and Interpreting Studies 19:2 ► pp. 234 ff. DOI logo
Lu, Rong, Muhammad Alif Redzuan Abdullah & Lay Hoon Ang
2023. Into-A or Into-B, That is a Question: A Systematic Literature Review of Directionality and Performance in Consecutive Interpreting. Sage Open 13:4 DOI logo
Lu, Xiaolei & Chao Han
2023. Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics. Interpreting. International Journal of Research and Practice in Interpreting 25:1 ► pp. 109 ff. DOI logo
Chen, Jing, Huabo Yang & Chao Han
2022. Holistic versus analytic scoring of spoken-language interpreting: a multi-perspectival comparative analysis. The Interpreter and Translator Trainer 16:4 ► pp. 558 ff. DOI logo
Yang, Yuan & Xiangdong Li
2022. Which theories are taught to students and how they are taught: A content analysis of interpreting textbooks . Círculo de Lingüística Aplicada a la Comunicación 92 ► pp. 167 ff. DOI logo
Han, Chao, Rui Xiao & Wei Su
2021. Assessing the fidelity of consecutive interpreting. Interpreting. International Journal of Research and Practice in Interpreting 23:2 ► pp. 245 ff. DOI logo
Wang, Weiwei
2021. Introducing China’s Standards of English Language Ability (CSE)—Interpreting Competence Scales. In Testing and Assessment of Interpreting [New Frontiers in Translation Studies, ], ► pp. 15 ff. DOI logo
Abdel Latif, Muhammad M. M.
2020. Translation and Interpreting Assessment Research. In Translator and Interpreter Education Research [New Frontiers in Translation Studies, ], ► pp. 61 ff. DOI logo
Wang, Weiwei, Yi Xu, Binhua Wang & Lei Mu
2020. Developing Interpreting Competence Scales in China. Frontiers in Psychology 11 DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue