Cover not available

Article published In: Interpreting
Vol. 24:1 (2022) ► pp.59–83

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Assessing spoken-language interpreting

The method of comparative judgement

ORCiD logo with link | Xiamen University
Published online: 1 October 2021
DOI logo with link
https://doi.org/10.1075/intp.00068.han
Abstract
In this study, we applied and evaluated a scoring method known as comparative judgement to assess spoken-language interpreting. This methodological exploration represents an extension of previous efforts to optimise scoring methods for assessing interpreting. Essentially, comparative judgement requires judges to compare two similar objects and make a binary decision about their relative qualities. To evaluate its reliability, validity and usefulness in the assessment of interpreting, we recruited two groups of judges (novice and experienced) to assess 66 two-way English/Chinese interpretations based on a computerised comparative judgement system. Our data analysis shows that the new method produced reliable and valid results across judge types and interpreting directions. However, the judges held polarised opinions about the method’s usefulness: while some considered it convenient, efficient and reliable, the opposite view was expressed by others. We discuss the results by providing an integrated analysis of the data collected, outline the perceived drawbacks and propose possible solutions to the drawbacks. We call for more evidence-based, substantive investigation into comparative judgement as a potentially useful method for assessing spoken-language interpreting in certain settings.
Article outline
  • 1.Introduction
  • 2.Comparative judgement
    • 2.1An introduction to comparative judgment
    • 2.2A comparative judgement approach to assessing spoken-language interpreting
    • 2.3Potential research gaps
  • 3.Research questions
  • 4.Method
    • 4.1Interpreting recordings
    • 4.2Participants
    • 4.3Online comparative judgment platform
    • 4.4Judge preparation and training
    • 4.5Procedures for comparative judgement
    • 4.6Post-hoc interview
    • 4.7Data analysis
  • 5.Results
    • 5.1Reliability evidence
    • 5.2Validity evidence
    • 5.3Judges’ perceived usefulness
  • 6.Discussion
    • 6.1Reliability
    • 6.2Validity
    • 6.3Perceived usefulness
    • 6.4Further analysis of potential drawbacks of comparative judgement
  • 7.Conclusion
  • Notes
  • References
References (45)
References
Andrich, D. (1978). Relationships between the Thurstone and Rasch approaches to item scaling. Applied Psychological Measurement 2 (3), 451–462. Google Scholar logo with link to Google Scholar
Barik, H. C. (1971). A description of various types of omissions, additions and errors of translation encountered in simultaneous interpretation. Meta 16 (4), 199–210. Google Scholar logo with link to Google Scholar
Bradley, R. A. & Terry, M. E. (1952). Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika 39 (3/4), 324–345. Google Scholar logo with link to Google Scholar
Bramley, T. (2015). Investigating the reliability of adaptive comparative judgement. [URL] (Accessed 9 June 2021).
Bramley, T., Bell, J. & Pollitt, A. (1998). Assessing changes in standards over time using Thurstone paired comparisons. Education Research and Perspectives 25 (2), 1–23.Google Scholar logo with link to Google Scholar
Bühler, H. (1986). Linguistic (semantic) and extralinguistic (pragmatic) criteria for the evaluation of conference interpretation and interpreters. Multilingua 51, 231–235.Google Scholar logo with link to Google Scholar
CCHI. (2012). Technical report on the development and pilot testing of the Certified Healthcare Interpreter™ (CHI™) examination for Arabic and Mandarin. [URL] (Accessed 9 June 2021).
Chen, J., Yang, H-B. & Han, C. (2021). Holistic versus analytic scoring of spoken-language interpreting: A multi-perspectival comparative analysis. Manuscript submitted for publication.Google Scholar logo with link to Google Scholar
(2017). Using analytic rating scales to assess English–Chinese bi-directional interpreting: A longitudinal Rasch analysis of scale utility and rater behaviour. Linguistica Antverpiensia, New Series: Themes in Translation Studies 161, 196–215. [URL]
(2019). A generalizability theory study of optimal measurement design for a summative assessment of English/Chinese consecutive interpreting. Language Testing 36 (3), 419–438. Google Scholar logo with link to Google Scholar
(2021). Interpreting testing and assessment: A state-of-the-art review. Language Testing. Google Scholar logo with link to Google Scholar
Hartley, A., Mason, I., Peng, G. & Perez, I. (2003). Peer- and self-assessment in conference interpreter training. Centre for Languages, Linguistics and Area Studies. [URL]
International School of Linguists. (2017). Diploma in Public Service Interpreting: Learner handbook. London, UK. [URL] (Accessed 9 June 2021).
Jones, I. & Inglis, M. (2015). The problem of assessing problem solving: Can comparative judgement help? Educational Studies in Mathematics 89 (3), 337–355. Google Scholar logo with link to Google Scholar
Jones, I. & Wheadon, C. (2015). Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation 471, 93–101. Google Scholar logo with link to Google Scholar
Jones, I., Swan, M. & Pollitt, A. (2015). Assessing mathematical problem solving using comparative judgement. International Journal of Science and Mathematics Education 131, 151–177. Google Scholar logo with link to Google Scholar
Laming, D. (2004). Marking university examinations: Some lessons from psychophysics. Psychology Learning and Teaching 3 (2), 89–96. Google Scholar logo with link to Google Scholar
Lee, J. (2008). Rating scales for interpreting performance assessment. The Interpreter and Translator Trainer 2 (2), 165–184. Google Scholar logo with link to Google Scholar
Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions 16 (2), 878.Google Scholar logo with link to Google Scholar
Liu, M. (2013). Design and analysis of Taiwan’s interpretation certification examination. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt: Peter Lang, 163–178.Google Scholar logo with link to Google Scholar
Luce, R. (1959). Individual choice behavior. New York: Wiley.Google Scholar logo with link to Google Scholar
McMahon, S. & Jones, I. (2015). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy & Practice 22 (3), 368–389. Google Scholar logo with link to Google Scholar
Myford, C. M. & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement 4 (4), 386–422.Google Scholar logo with link to Google Scholar
National Center for State Courts. (2019). Federal Court Interpreter Certification Examination for Spanish/English: Examinee handbook. [URL] (Accessed 9 June 2021).
Pöchhacker, F. (2002). Researching interpreting quality: Models and methods. In G. Garzone & M. Viezzi (Eds.), Interpreting in the 21st century: Challenges and opportunities. Amsterdam: John Benjamins, 95–106. Google Scholar logo with link to Google Scholar
Pollitt, A. (2012a). Comparative judgement for assessment. International Journal of Technology and Design Education 22 (2), 157–170. Google Scholar logo with link to Google Scholar
(2012b). The method of adaptive comparative judgement. Assessment in Education: Principles, Policies & Practice 19 (3), 281–300.Google Scholar logo with link to Google Scholar
Pollitt, A. & Murray, N. L. (1996). What raters really pay attention to? In M. Milanovic, & N. Saville (Eds.), Studies in language testing 3: Performance testing, cognition and assessment. Cambridge: Cambridge University Press, 74–91.Google Scholar logo with link to Google Scholar
PSI Services LLC. (2013). Development and validation of oral examinations for medical interpreter certification: Mandarin, Russian, Cantonese, Korean, and Vietnamese forms. [URL] (Accessed 9 June 2021).
Riccardi, A. (1998). Evaluation in interpretation: Macrocriteria and microcriteria. In E. Hung (Ed.), Teaching translation and interpreting 4. Building bridges. Amsterdam: John Benjamins, 115–127. Google Scholar logo with link to Google Scholar
Roberts, R. P. (2000). Interpreter assessment tools for different settings. In R. P. Roberts, S. E. Carr, D. Abraham & A. Dufour (Eds.), Critical link 2: Interpreters in the community. Amsterdam: John Benjamins, 103–130. Google Scholar logo with link to Google Scholar
Salkind, N. J. (2007). Encyclopedia of measurement and statistics. Thousand Oaks, CA: Sage. Google Scholar logo with link to Google Scholar
Schjoldager, A. (1996). Assessment of simultaneous interpreting. In C. Dollerup & V. Appel (Eds.), Teaching translation and interpreting 3: New horizons. Amsterdam: John Benjamins, 187–195. Google Scholar logo with link to Google Scholar
Setton, R. & Dawrant, A. (2016). Conference interpreting: A trainer’s guide. Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review 34 (4), 273–286. Google Scholar logo with link to Google Scholar
(1954). The measurement of values. Psychological Review 61 (1), 47–58. Google Scholar logo with link to Google Scholar
Tiselius, E. (2009). Revisiting Carroll’s scales. In C. V. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies. Amsterdam: John Benjamins, 95–121. Google Scholar logo with link to Google Scholar
Verhavert, S., Bouwer, R., Donche, V. & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice 26 (5), 541–562. Google Scholar logo with link to Google Scholar
Wu, S. (2010). Assessing simultaneous interpreting: A study on test reliability and examiners’ assessment behavior. PhD thesis, Newcastle University.
(2013). How do we assess students in the interpreting examinations? In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt am Main: Peter Lang, 15–33.Google Scholar logo with link to Google Scholar
Cited by (8)

Cited by eight other publications

Han, Chao & Yueqing Wang
2025. Conducting replication in translation and interpreting studies. Target. International Journal of Translation Studies 37:3 ► pp. 444 ff. DOI logo
Sickinger, Rebecca, Tineke Brunfaut & John Pill
2025. Comparative Judgement for evaluating young learners’ EFL writing performances: Reliability and teacher perceptions of holistic and dimension-based judgements. Language Testing 42:2 ► pp. 137 ff. DOI logo
Wang, Jingwen & Ying Zheng
2025. Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement. Language Testing DOI logo
Guo, Wei, Xun Guo, Junkang Huang & Sha Tian
2024. Modeling listeners’ perceptions of quality in consecutive interpreting: a case study of a technology interpreting event. Humanities and Social Sciences Communications 11:1 DOI logo
Thwaites, Peter, Charalambos Kollias & Magali Paquot
2024. Is CJ a valid, reliable form of L2 writing assessment when texts are long, homogeneous in proficiency, and feature heterogeneous prompts?. Assessing Writing 60 ► pp. 100843 ff. DOI logo
Thwaites, Peter & Magali Paquot
2024. Comparative judgement for advancing research in applied linguistics. Research Methods in Applied Linguistics 3:3 ► pp. 100142 ff. DOI logo
Lu, Xiaolei & Chao Han
2023. Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics. Interpreting. International Journal of Research and Practice in Interpreting 25:1 ► pp. 109 ff. DOI logo
Han, Chao, Bei Hu, Qin Fan, Jing Duan & Xi Li
2022. Using computerised comparative judgement to assess translation. Across Languages and Cultures 23:1 ► pp. 56 ff. DOI logo

This list is based on CrossRef data as of 12 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue