The reception of translated vaccination information: Evidence from a reading and stops-making-sense judgment task
Reception-oriented research on health communication, especially when focusing on migrant populations, allows for an exploration of what it means to provide access to health information, shedding light on migrants’ communicative needs with practical implications for translators’ work. Adopting a multi-method approach, this study explores the potential for reading difficulty and for misinterpretation of translated vaccination-related public health information. For this purpose, we conducted a reception-oriented study that used a participant-paced, region-by-region reading method coupled with an incremental stops-making-sense judgment task and a post-hoc comprehension questionnaire with two participant groups (native and non-native English speakers). Three main findings stand out from our analysis. First, the triangulation of multiple data sources indicates that reading difficulty and misinterpretation were not exclusive to the non-native group but affected both participant groups. Second, while participants generally responded correctly to comprehension questions, average reading times showed that both groups took longer to read most texts than expected. Third, medical language was the most challenging aspect for our participants, as indicated by data from online and offline subjective measures.
Publication history
Table of contents
- Abstract
- Keywords
- 1.Introduction
- 2.Related research
- 3.Methods and materials
- 4.Results
- 5.Conclusion
- 1.Introduction
- 2.Related research
- 3.Methods and materials
- 4.Results
- 5.Conclusion
- Acknowledgements
- Notes
- Funding
- Acknowledgements
- Notes
- References
- Address for correspondence
1.Introduction
The reception of translated texts, and in particular, of health information, is an under-researched topic in Cognitive Translation and Interpreting Studies (CTIS). Contemporary societies are increasingly multilingual and multicultural due to several factors, including mobility and migration (UNHCR 2023UNHCR 2023 Global Trends: Forced Displacement in 2022. Copenhagen: UNHCR. https://www.unhcr.org/sites/default/files/2023-06/global-trends-report-2022.pdf; GMDAC n.d.GMDAC n.d. “Forced Migration or Displacement.” Accessed June 28, 2023. https://www.migrationdataportal.org/themes/forced-migration-or-displacement). Effective communication with diverse groups within society, especially those that might be exceptionally vulnerable, is crucial, particularly in healthcare contexts. Providing translated health information in public communication demands an evidence-based understanding of multilingual communities and their communicative needs. This is where reception-oriented research on health communication, especially when focusing on migrant populations, can make a meaningful contribution. It allows us to explore what it means to provide access to health information, shedding light on migrants’ communicative needs with practical implications for translators’ work.
Despite the relevance of readership and reception for translation, our understanding of how readers read and comprehend translated texts remains limited (Walker 2021 2021 “Investigating How We Read Translations: A Call to Action for Experimental Studies of Translation Reception.” Cognitive Linguistic Studies 8 (2): 482–512. ). This is particularly evident in the case of written translations, where research lags behind that of audiovisual translation. In CTIS, research dedicated to readers’ reception of written translations is comparatively limited (Kruger and Kruger 2017Kruger, Haidee, and Jan-Louis Kruger 2017 “Cognition and Reception.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 71–89. Malden: Wiley-Blackwell. ), despite the potential of cognitive research to shed light on the process of comprehension and meaning-making (a notable exception is Kruger 2013 2013 “Child and Adult Readers’ Processing of Foreign Elements in Translated South African Picturebooks: An Eye-Tracking Study.” Target 25 (2): 180–227. ). This indicates an opportunity to advance our understanding of how readers process translated texts, especially in contexts such as health and vaccination, where their interpretation might have behavioral consequences (i.e., a secondary effect; Chesterman 1998Chesterman, Andrew 1998 “Causes, Translations, Effects.” Target 10 (2): 201–230. ).
To tackle this, we set out to investigate lay readers’ cognitive processing of authentic translated medical texts by triangulating online and offline data collection methods (for the distinction between online and offline methods, see Krings [2005]Krings, Hans Peter 2005 “Wege ins Labyrinth — Fragestellungen und Methoden der Übersetzungsprozessforschung im Überblick [Paths into the labyrinth — An overview of the questions and methods of research into the translation process].” Meta 50 (2): 342–358. ). This initial exploratory study is the first step in a larger research project investigating readers’ cognitive effort when reading translated health information. Without empirical data on reader reception, translators’ decisions about how to translate would remain largely speculative, relying on individual assumptions rather than evidence-based insights into readers’ needs and expectations (Brems and Ramos Pinto 2013Brems, Elke, and Sara Ramos Pinto 2013 “Reception and Translation.” In Handbook of Translation Studies: Volume 4, edited by Yves Gambier and Luc van Doorslaer, 142–147. Amsterdam: John Benjamins. , 145). Reception-oriented research can provide valuable data to inform translators and translation training, allowing translators to make more informed decisions that align with the needs of their target audience.
Using the participant-paced, region-by-region reading method, accompanied by an incremental stops-making-sense judgment task and a post-hoc comprehension questionnaire, we explore the potential for reading difficulty and for misinterpretation of translated vaccination-related public health information. To understand the differences between native and non-native speakers, we further test if comprehension difficulty and misinterpretation differ across two groups with different language profiles. For this purpose, we invited native Spanish speakers with English as a second language and currently living in the Netherlands to read translated vaccination-related information that forms part of official translations into English published on the website of the Dutch Health Ministry. The Spanish group’s data were compared with those of native English speakers living in the Netherlands.
We considered it important to conduct this exploratory reception study with official translated information due to the impact that this public health information may have, especially on migrants who may rely, sometimes exclusively, on this translated information to make health decisions that affect them and their families. It is worth acknowledging that the translations used in this study may have been carried out by professional or non-professional/informal translators, that part of their brief might have included specific instructions to use intralingual strategies, and that different translation technologies may have been used.
To design the study, we drew on the theoretical and methodological approaches and related lessons from various and often scattered studies within cognitive research that have employed multimethod approaches to study processes and reception in translation and interpreting.
Before describing the study design, we review experimental research on the reception of translated texts (Section 2.1) and a selection of studies explicitly focusing on translated health information (Section 2.2). The methods are described in Section 3, and the results are reported in Section 4. In the last section, we discuss the findings, drawing conclusions encompassing the study’s implications and limitations, and outlining potential for future research.
2.Related research
While research in the field of CTIS on how viewers experience and respond to audiovisual products has been particularly prolific, research on the reception of written translated texts has been scarce, and that is the focus of our review. For an overview of the literature on the reception of audiovisual products see, for instance, Kruger (2018)Kruger, Jan-Louis 2018 “Eye Tracking in Audiovisual Translation Research.” In The Routledge Handbook of Audiovisual Translation, edited by Luis Pérez-González, 350–366. London: Routledge. and Di Giovanni and Gambier (2018)Di Giovanni, Elena, and Yves Gambier eds. 2018 Reception Studies and Audiovisual Translation. Amsterdam: John Benjamins. , in particular Chapter 5 (Kruger and Doherty 2018Kruger, Jan-Louis, and Stephen Doherty 2018 “Triangulation of Online and Offline Measures of Processing and Reception in AVT.” In Reception Studies and Audiovisual Translation, edited by Elena Di Giovanni and Yves Gambier, 91–110. Amsterdam: John Benjamins. ).
2.1Experimental research on the reception of translated texts
Literature on reading translated texts has mainly focused on how translators read texts with a specific aim: reading for translation and other related tasks (Kruger and Kruger 2017Kruger, Haidee, and Jan-Louis Kruger 2017 “Cognition and Reception.” In The Handbook of Translation and Cognition, edited by John W. Schwieter and Aline Ferreira, 71–89. Malden: Wiley-Blackwell. , 71). Shreve and colleagues (1993)Shreve, Gregory M., Christina Schäffner, Joseph H. Danks, and Jennifer Griffin 1993 “Is There a Special Kind of ‘Reading’ for Translation? An Empirical Investigation of Reading in the Translation Process.” Target 5 (1): 21–41. were among the first to examine the role of reading in translation, comparing reading for translation, paraphrasing, and comprehension. Building on Jakobsen and Jensen’s (2009)Jakobsen, Arnt Lykke, and Kristian Jensen 2009 “Eye Movement Behaviour across Four Different Types of Reading Task.” Copenhagen Studies in Language 36: 103–124. well-known study, Alves and colleagues (2011)Alves, Fabio, Adriana Pagano, and Igor Da Silva 2011 “Towards an Investigation of Reading Modalities in/for Translation: An Exploratory Study Using Eye-Tracking Data.” In Cognitive Explorations of Translation, edited by Sharon O’Brien, 175–196. London: Continuum. examined reading differences for reading for comprehension, producing an oral summary, and sight translation. Similarly, Hvelplund (2017)Hvelplund, Kristian Tangsgaard 2017 “Four Fundamental Types of Reading during Translation.” In Translation in Transition: Between Cognition, Computing and Technology, edited by Arnt Lykke Jakobsen and Bartolomé Mesa-Lao, 55–77. Amsterdam: John Benjamins. investigated the differences between various types of reading: source text reading, source text reading while typing, reading of existing target text, and emerging target text. One shared aspect among these studies is the suggestion that the reading purpose and task objective influence reading processes. Hence, there is a need for research that investigates explicitly the reading of translated texts by non-translators/readers.
Despite the “centrality” of readers’ reception of translated texts for translation research, as Walker (2021 2021 “Investigating How We Read Translations: A Call to Action for Experimental Studies of Translation Reception.” Cognitive Linguistic Studies 8 (2): 482–512. , 485) explains, there has been very little experimental research on the topic. Kruger’s (2012Kruger, Haidee 2012 Postcolonial Polysystems: The Production and Reception of Translated Children’s Literature in South Africa. Amsterdam: John Benjamins. , 2013 2013 “Child and Adult Readers’ Processing of Foreign Elements in Translated South African Picturebooks: An Eye-Tracking Study.” Target 25 (2): 180–227. ) study on translated South African picture books is one of the first and most cited reading studies in CTIS. Using eye-tracking, questionnaire, and interview data, it compares how child and adult readers process and respond to foreignized and domesticated elements, focusing on the cognitive effort involved in reading and how comprehension is affected by these foreignized elements.
Similarly, Walker (2019)Walker, Callum 2019 “A Cognitive Perspective on Equivalent Effect: Using Eye Tracking to Measure Equivalence in Source Text and Target Text Cognitive Effects on Readers.” Perspectives 27 (1): 124–143. examined readers’ experiences of stylistic language varieties in literature. Using eye-tracking data and focusing on the notion of equivalent effect, he compared the reading behavior of native readers of a French literary source text and native readers of its English translation. The study shows variations in cognitive effort between the two groups suggesting that equivalent effect was not always achieved.
More recently, Whyatt and colleagues (2023)Whyatt, Bogusława, Olga Witczak, Ewa Tomczak-Łukaszewska, and Olha Lehka-Paul 2023 “The Proof of the Translation Process Is in the Reading of the Target Text: An Eyetracking Reception Study.” Ampersand 11: 100149. reported on the first results of the Read Me Project, aiming to compare the reading experience of translated texts with different levels of quality using eye-tracking and questionnaires. Their first study focused on how the translator’s effort, as manifested in translation process data, is reflected in the reader’s effort, showing that the reader’s reception effort is “significantly affected” (2023, 7) by the quality of the target text.
A different perspective, but one worth noting, is the growing body of research on the comprehensibility and/or acceptability of machine-translated texts. These studies use methods such as eye-tracking and retrospective think-aloud, often in combination with questionnaires. A few examples are Doherty and O’Brien (2012Doherty, Stephen, and Sharon O’Brien 2012 “A User-Based Usability Assessment of Raw Machine Translated Technical Instructions.” In Proceedings of the 10th Conference of the Association for Machine Translation in the Americas: Commercial MT User Program. Association for Machine Translation in the Americas., 2014 2014 “Assessing the Usability of Raw Machine Translated Output.” International Journal of Human-Computer Interaction 30 (1): 40–51. ); Guerberof Arenas, Moorkens, and O’Brien (2019)Guerberof Arenas, Ana, Joss Moorkens, and Sharon O’Brien 2019 “What Is the Impact of Raw MT on Japanese Users of Word: Preliminary Results of a Usability Study Using Eye-Tracking.” Proceedings of Machine Translation Summit XVII Volume 1: Research Track 1: 67–77.; and Screen (2019)Screen, Benjamin 2019 “What Effect Does Post-Editing Have on the Translation Product from an End-User’s Perspective?” JoSTrans 31: 133–157.. These studies find sometimes significant differences in usability between source and machine-translated outputs in different language pairs.
2.2Reception-oriented medical translation research
In recent years, there has been a surge of interest in researching the translation of health information (or medical translation), and a new wave of publications has emerged that is specifically focused on two different but, in our view, connected trends: the translation of information, including health-related information, during a crisis (what is now known as ‘crisis translation’); and the translation of public health information in general (i.e., not necessarily in the context of a health-related crisis). For an overview of the research conducted on crisis translation, see O’Brien (2022)O’Brien, Sharon 2022 “Crisis Translation: A Snapshot in Time.” INContext: Studies in Translation and Interculturalism 2 (1): 84–108. and the Interact website.11.See https://sites.google.com/view/crisistranslation/home.
This trend has gained momentum in the wake of the Covid-19 pandemic, leading to a proliferation of research projects (and associated publications) on the translation of health-related information, such as “Towards an Inclusive Crisis Communication Policy” (Vandenbroucke et al. 2022Vandenbroucke, Mieke, Nina Reviers, Gert Vercauteren, et al 2022 Final Report: Towards an Inclusive Covid-19 Crisis Communication Policy in Belgium. Antwerpen: Universiteit Antwerpen. https://hdl.handle.net/10067/1950130151162165141) or “COVID-19 Recovery” (British Academy 2022British Academy 2022 Overcoming Barriers to Vaccination by Empowering Citizens to Make Deliberate Choices. London: The British Academy. https://www.thebritishacademy.ac.uk/international/affiliations/ssh7/covid-19-vaccine-engagement-in-the-uk-and-usa/overcoming-barriers-vaccination-empowering-citizens).22.While in September 2018, the keyword ‘medical discourse’ (or ‘medicine’, ‘medical sciences’) in the online Translation Studies Bibliography retrieved 487 publications, in April 2023 it retrieved 623 publications, indicating a clear increase in the number of publications. The word ‘Covid-19’ alone retrieved 93 publications in April 2023. For a review of the role of medical translation and interpreting within Translation Studies before 2010, see Franco Aixelá (2010)Franco Aixelá, Javier 2010 “Una revisión de la bibliografía sobre traducción e interpretación médica recogida en BITRA (Bibliografía de Interpretación y Traducción) [An overview of the bibliography related to medical translation and interpreting as collected in BITRA (Bibliography of Interpreting and Translation)].” Panace@ 11 (32): 151–160..
Despite this renewed interest in the translation of health-related topics, our understanding of how translated public health information or translated medical texts are received by readers is still limited. In the following paragraphs, we review a selection of empirical studies on the reception of translated health information.
Studies adopting a reception-oriented approach on how laypeople receive translated medical content have mostly adopted offline methods and focused on specific text types. Askehave and Zethsen (2003 2003 “Communication Barriers in Public Discourse.” Information Design Journal 4 (1): 23–41., 2014 2014 “A Comparative Analysis of the Lay-Friendliness of Danish EU Patient Information Leaflets from 2000 to 2012.” Communication and Medicine 11 (3): 209–222. ), for example, conducted questionnaire studies on the lay-friendliness of translated package inserts in Danish. They suggest that medical information is usually challenging for laypeople to understand (Askehave and Zethsen 2000Askehave, Inger, and Karen Korning Zethsen 2000 “Medical Texts Made Simple — Dream or Reality?” HERMES — Journal of Language and Communication in Business 13 (25): 63–74. ; Brøgge and Zethsen 2021Brøgge, Matilde Nisbeth, and Karen Korning Zethsen 2021 “Inter- and Intralingual Translation of Medical Information.” In The Routledge Handbook of Translation and Health, edited by Şebnem Susam-Saraeva and Eva Spišiaková, 96–107. London: Routledge. ). The principal factors contributing to this include specialized medical language, characterized by lengthy and intricate sentences, nominalization, extensive use of passive voice, and the inclusion of excessive information within a single sentence. Specific terminology, mainly Latin-Greek terms, poses a further obstacle to comprehension.
The studies conducted by García-Izquierdo and Muñoz-Miquel (2015)García-Izquierdo, Isabel, and Ana Muñoz-Miquel 2015 “Los folletos de información oncológica en contextos hospitalarios [Cancer information leaflets in hospitals: Perspectives of patients and healthcare professionals].” Panace@ 16 (42): 225–231. and García-Izquierdo (2016)García-Izquierdo, Isabel 2016 “At the Cognitive and Situational Interface: Translation in Healthcare Settings.” Translation Spaces 5 (1): 20–37. as part of the MedGentt project employed questionnaires, interviews, and focus groups to examine the needs of health professionals and patients together with the suitability and readability of written resources available in hospitals. Their findings are consistent with those of Askehave and Zethsen (2003 2003 “Communication Barriers in Public Discourse.” Information Design Journal 4 (1): 23–41., 2014 2014 “A Comparative Analysis of the Lay-Friendliness of Danish EU Patient Information Leaflets from 2000 to 2012.” Communication and Medicine 11 (3): 209–222. ), as they conclude that there is a need to reduce the level of technicality and facilitate comprehension by implementing translation strategies such as reformulation and recontextualization. They argue that this approach could improve adherence to the therapeutic plan (García-Izquierdo and Muñoz-Miquel 2015García-Izquierdo, Isabel, and Ana Muñoz-Miquel 2015 “Los folletos de información oncológica en contextos hospitalarios [Cancer information leaflets in hospitals: Perspectives of patients and healthcare professionals].” Panace@ 16 (42): 225–231.).
O’Brien and Cadwell (2017)O’Brien, Sharon, and Patrick Cadwell 2017 “Translation Facilitates Comprehension of Health-Related Crisis Information: Kenya as an Example.” JoSTrans 28: 23–51. tested the comprehension of health-related crisis communication among Kenyans. The study used a survey that asked participants to read an information poster about Ebola in English and another group to read a translated version of the poster in Kiswahili, followed by comprehension questions. The findings emphasize the need to translate public health information into Kiswahili and underscore that English is inadequate for conveying crucial information in this context.
To our knowledge, there has yet to be an attempt to study the reception of translated vaccination-related information using offline and online methods.
3.Methods and materials
As part of a larger research project aiming to explore lay readers’ cognitive processing of medical translated texts, we conducted a quasi-experiment focusing on the potential for reading difficulty and for misinterpretation of translated vaccination-related public health information.33.In natural reading studies, such as this, participants are not randomly allocated to one of the experimental groups (see Mellinger and Hanson 2017Mellinger, Christopher, and Thomas A. Hanson 2017 Quantitative Research Methods in Translation and Interpreting Studies. London: Routledge., 7–8 for an explanation of when experiments can be considered ‘quasi’). The data and findings of this experiment will serve as a stepping-stone for a follow-up eye-tracking experiment that aims to look at readers’ cognitive effort involved in reading translated health information.
The following sections describe the methods and materials. This description is as detailed as possible to allow for future replication. For the materials, see https://doi.org/10.5281/zenodo.13605400. Whenever possible, we make explicit the ethical considerations and the reasoning behind our decisions that affect the study’s design. These considerations follow the best practices in research as discussed by scholars in Translation Studies and CTIS (e.g., Saldanha and O’Brien 2013Saldanha, Gabriela, and Sharon O’Brien 2013 Research Methodologies in Translation Studies. Abingdon: Routledge.; Mellinger and Hanson 2017Mellinger, Christopher, and Thomas A. Hanson 2017 Quantitative Research Methods in Translation and Interpreting Studies. London: Routledge., 2022 2022 “Considerations of Ecological Validity in Cognitive Translation and Interpreting Studies.” Translation, Cognition & Behavior 5 (1): 1–26. ; Mellinger and Baer 2021Mellinger, Christopher, and Brian James Baer 2021 “Research Ethics in Translation and Interpreting Studies.” In Routledge Handbook of Translation and Ethics, edited by Kaisa Koskinen and Nike K. Pokorn, 365–380. New York: Routledge.).
3.1Participants
The study consisted of two groups of participants: a group of native English speakers (n = 11) and a group of native Spanish speakers with English as their second language (n = 11). We did not exclude speakers of any variant. To establish the Spanish group’s language proficiency in English, participants were asked to complete three of the BEST subtests (see Section 3.2). Table 1 gives an overview of the participants.
| Categories | English group | Spanish group |
|---|---|---|
| Gender | ||
| Female | 10 | 6 |
| Male | 1 | 5 |
| Age | ||
| 18–34 | 4 | 5 |
| 35–54 | 6 | 6 |
| 55–64 | 1 | 0 |
| Education | ||
| BA | 7 | 2 |
| MA | 3 | 7 |
| PhD | 1 | 2 |
To recruit the participants and target, as much as possible, migrants from outside our personal networks, we posted calls for participation within dedicated groups of migrants living in the Netherlands using social media, WhatsApp, newsletters, and networks of Dutch universities.
As per the criteria for inclusion, all participants were eighteen years or older and were currently living in the Netherlands. Regarding reading habits, all reported English as their preferred language for reading for leisure and/or work. Except for two native English speakers, all participants reported being used to reading scientific/technical texts for work and/or leisure.
3.2English language proficiency of the native Spanish speakers
Since there is a risk of participants over- or underestimating their language proficiency (MacIntyre, Noels, and Clément 1997MacIntyre, Peter D., Kimberly A. Noels, and Richard Clément 1997 “Biases in Self-Ratings of Second Language Proficiency: The Role of Language Anxiety.” Language Learning 47 (2): 265–287. ), it is not advisable to depend solely on self-assessments as a gauge of language usage and proficiency, especially when that proficiency is an important variable for a given study (e.g., Tiselius and Sneed 2020Tiselius, Elisabet, and Kayle Sneed 2020 “Gaze and Eye Movement in Dialogue Interpreting: An Eye-Tracking Study.” Bilingualism: Language and Cognition 23 (4): 780–787. , 782). Given this, the English proficiency level of the group of Spanish speakers was assessed based on part of the BEST dataset (De Bruin, Carreiras, and Duñabeitia 2017De Bruin, Angela, Manuel Carreiras, and Jon Andoni Duñabeitia 2017 “The BEST Dataset of Language Proficiency.” Frontiers in Psychology 8. ), which includes a multi-dimensional battery of tests.
The original BEST dataset consists of four subtests: (1) a vocabulary test, based on the Multilingual Naming Test (MINT) (Gollan et al. 2012Gollan, Tamar H., Gali H. Weissberger, Elin Runnqvist, Rosa I. Montoya, and Cynthia M. Cera 2012 “Self-Ratings of Spoken Language Dominance: A Multilingual Naming Test (MINT) and Preliminary Norms for Young and Aging Spanish–English Bilinguals.” Bilingualism: Language and Cognition 15 (3): 594–615. ); (2) a lexical decision test, based on the Lexical Test for Advanced Learners of English (LexTALE) (Lemhöfer and Broersma 2012Lemhöfer, Kristin, and Mirjam Broersma 2012 “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–343. ); (3) a short semi-structured interview; and (4) a short questionnaire about language history and knowledge. Considering the purposes of this study and our limitations in terms of time and resources, we incorporated Tests 1, 2, and 4 in a pre-task questionnaire, and based on these data, the native Spanish speakers’ proficiency level in English was assessed (see Table 2).
LexTALE scores were calculated based on Lemhöfer and Broersma (2012Lemhöfer, Kristin, and Mirjam Broersma 2012 “Introducing LexTALE: A Quick and Valid Lexical Test for Advanced Learners of English.” Behavior Research Methods 44 (2): 325–343. , 241), and MINT scores on Tao, Taft, and Gollan (2015)Tao, Lily, Marcus Taft, and Tamar H. Gollan 2015 “The Bilingual Switching Advantage: Sometimes Related to Bilingual Proficiency, Sometimes Not.” Journal of the International Neuropsychological Society 21 (7): 531–544. ; and Lizarazu et al. (2021)Lizarazu, Mikel, Manuel Carreiras, Mathieu Bourguignon, Asier Zarraga, and Nicola Molinaro 2021 “Language Proficiency Entails Tuning Cortical Activity to Second Language Speech.” Cerebral Cortex 31 (8): 3820–3831. .
| Participant number | Self-rated proficiency in English | MINT | LexTALE | |
|---|---|---|---|---|
| Tao, Taft, and Gollan (2015)Tao, Lily, Marcus Taft, and Tamar H. Gollan 2015 “The Bilingual Switching Advantage: Sometimes Related to Bilingual Proficiency, Sometimes Not.” Journal of the International Neuropsychological Society 21 (7): 531–544. | Lizarazu et al. (2021)Lizarazu, Mikel, Manuel Carreiras, Mathieu Bourguignon, Asier Zarraga, and Nicola Molinaro 2021 “Language Proficiency Entails Tuning Cortical Activity to Second Language Speech.” Cerebral Cortex 31 (8): 3820–3831. | Lemhöfer and Broersma (2012) | ||
| 203 | Advanced user (B2) | High | C1 | B2 |
| 204 | Advanced user (B2) | High | C1 | B2 |
| 205 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
| 206 | Proficient user (C1 or C2) | High | C1 | B2 |
| 208 | Proficient user (C1 or C2) | High | C1 | B2 |
| 209 | Proficient user (C1 or C2) | High | C1 | B2 |
| 210 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
| 213 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
| 214 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
| 215 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
| 217 | Proficient user (C1 or C2) | High | C1 | C1 or C2 |
Of the Spanish group, nine participants reported that they were Proficient users of English (equivalent to C1 or C2)44.According to the Common European Framework of Reference for Languages (CEFR; see https://www.coe.int/en/web/common-european-framework-reference-languages/level-descriptions). and two Advanced users (equivalent to B2). According to the MINT test, all eleven participants have a high or C1 level. The LexTALE test assessed five participants at a B2 level and the remaining at a C1 or C2 level. This means that the Spanish-speaking participants in our experiment have a high level of English proficiency, ranging between B2 and C2, depending on the test.
3.3Tasks and procedure
The data collection took place between December 2022 and March 2023, following a pilot study with five Spanish and native English speakers. Prior to the data collection, all participants read an information letter and signed an informed consent form in their preferred language (that is, English or Spanish). This procedure is in accordance with the Guidelines of the Ethics Committee of Faculties of Humanities and Archaeology of Leiden University from which this study received ethical approval (Reference 2022/26). Participants were given the opportunity to ask questions about the study, their participation, and how their data would be handled before giving consent, as well as during and after the data collection. By adopting this practice, we view informed consent as an ongoing and iterative process that prioritizes participants and provides ample opportunity to address their questions and concerns (Mellinger and Baer 2021Mellinger, Christopher, and Brian James Baer 2021 “Research Ethics in Translation and Interpreting Studies.” In Routledge Handbook of Translation and Ethics, edited by Kaisa Koskinen and Nike K. Pokorn, 365–380. New York: Routledge., 371). Following our data management plan, data collection, management, and storage procedures complied with the EU General Data Protection Regulation. Participants were remunerated a standard participant fee.
3.3.1The steps of the procedure
The procedure consisted of three steps.
Step 1: Pre-reading task questionnaire (demographics, reading habits, and English proficiency level)
Native speakers of Spanish and English were invited to answer an online pre-reading task questionnaire designed in Qualtrics and consisting of five sections (see Figure 1). Native Spanish speakers were additionally assessed on their proficiency level in English.
Step 2: Participant-paced, region-by-region reading task and incremental stops-making-sense judgment task
After the first questionnaire, participants were invited to the lab, where they were asked to read seventeen target texts (three of which were warm-up texts) presented in a region-by-region fashion, following the self-paced reading moving window paradigm in which readers cannot see the text they just read or the text that comes ahead (Just, Carpenter, and Woolley 1982Just, Marcel A., Patricia A. Carpenter, and Jacqueline D. Woolley 1982 “Paradigms and Processes in Reading Comprehension.” Journal of Experimental Psychology: General 111 (2): 228–238. ). We coupled this task with an adaptation of the incremental judgment task (Roland et al. 2012Roland, Douglas, Hongoak Yun, Jean-Pierre Koenig, and Gail Mauner 2012 “Semantic Similarity, Predictability, and Models of Sentence Processing.” Cognition 122 (3): 267–279. ), also referred to as a ‘stops-making-sense’ task (Müller and Mari 2021Müller, Misha-Laura, and Magali A. Mari 2021 “Definite Descriptions in the Light of the Comprehension vs Acceptance Distinction: Comparing Self-Paced Reading with Eye-Tracking Measures.” Frontiers in Communication 6: 634362. ). In stops-making-sense tasks, participants are instructed to press a key to continue making words appear, region by region, as long as the sentence makes sense to them or as long they keep understanding the text (Singh et al. 2016Singh, Raj, Evelina Fedorenko, Kyle Mahowald, and Edward Gibson 2016 “Accommodating Presuppositions Is Inappropriate in Implausible Contexts.” Cognitive Science 40 (3): 607–634. , 615). Once a participant encounters a new region that no longer makes sense to them in the context of the text, they are instructed to stop the task by pressing a different key. In traditional stops-making-sense tasks, participants would then move to the next reading task, interrupting the current reading. In our experiment, participants could continue reading. This modification allowed readers to finish reading the text as they probably would outside a lab experiment because we were interested in identifying the reading difficulty of different regions in the same text. These tasks were run in PsychoPy v2022.1.4.
Step 3: Post-hoc comprehension questionnaire
Participants were also asked to complete an online comprehension questionnaire built in Qualtrics. It consisted of re-reading the same seventeen texts as in the previous reading task. Each text was followed by two questions, one more specific, which targeted the comprehension of a specific region, and the second more general, which targeted the global understanding of the text (see Figure 2).
Tetanus
Tetanus leads to violent muscle spasms. Without treatment, tetanus is fatal. Tetanus is not contagious. That means that it cannot spread from person to person. A child can get tetanus after a bite from a pet or other animal, or if dirt from the street gets into an open wound.
*A person can catch tetanus from another person.
○ I don’t know
○ False
○ True
*What is the main message of this text? Select the best option.
○ Tetanus causes death when it is not treated.
○ Tetanus may cause death when it is not treated.
○ Tetanus only causes death in animals.
○ I don’t know.
All questions were single or multi-select multiple choice and included an ‘I don’t know’ option that participants were instructed to select in case they were not sure what the correct answer was. The order of the options within each question was automatically randomized. At the end, participants were asked to rate the difficulty of the texts on a five-point Likert scale, followed by an open question that asked, “What did you find most difficult to understand in these texts?”
3.4Target texts
The reading task included seventeen texts, three of which were practice texts. The texts were selected based on two conditions. Given the research aim, the target texts had to be (1) health-related and (2) official translations into English published by the Dutch National Institute for Public Health and the Environment (‘Rijksinstituut voor Volksgezondheid en Milieu’; RIVM).55.See https://www.rivm.nl/en. The premise was that the official English translated version made available by the Dutch Health Ministry would be one of the primary official sources of information for migrants in the Netherlands interested in accessing health information.
Target texts were selected from the National Immunization Program brochure. This brochure, explicitly identified as a translation under the section “Translations of Letters and Brochures” of the RIVM website, was first made available in December 2021 and aims to inform how vaccinations protect children against infectious diseases. This brochure targets parents, and the selected texts describe the national vaccination program’s aims and briefly explain twelve infectious diseases covered by the program. This same brochure is also included in the recently created health information page aimed at refugees from Ukraine in the Netherlands, available since December 2022. Table 3 displays the text metrics of the target texts used in the reading task, excluding the practice texts.66.While the texts are presented in the order they appear in the published brochure, the reading order was randomized.
| Text | 1 Protect your child | 2 Healthy start in life | 3 Diphtheria | 4 Whooping cough | 5 Tetanus | 6 Polio | 7 Hib disease | 8 Hepatitis B | 9 Pneumococcal disease | 10 Mumps | 11 Measles | 12 Rubella | 13 Meningococcal disease | 14 HPV |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Word count | 89 | 50 | 16 | 72 | 51 | 13 | 19 | 25 | 40 | 30 | 36 | 53 | 81 | 66 |
| Number of sentences | 5 | 6 | 3 | 6 | 6 | 2 | 2 | 2 | 3 | 3 | 3 | 4 | 6 | 4 |
| Average words per sentence | 17.80 | 8.3 | 5.3 | 12 | 8.5 | 7 | 9.5 | 12.5 | 13.3 | 10 | 12 | 13.2 | 13.5 | 16.5 |
| Number of characters (with spaces) | 589 | 309 | 112 | 478 | 287 | 88 | 159 | 143 | 282 | 185 | 223 | 331 | 577 | 401 |
| 1 Protect your child | 2 Healthy start in life | 3 Diphtheria | 4 Whooping cough | 5 Tetanus | 6 Polio | 7 Hib disease | 8 Hepatitis B | 9 Pneumococcal disease | 10 Mumps | 11 Measles | 12 Rubella | 13 Meningococcal disease | 14 HPV | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Flesch Reading Ease | 39 | 62 | 32 | 45 | 70 | 43 | 0 | 49 | 22 | 61 | 47 | 45 | 21 | 45 |
| Flesh-Kincaid Grade Level | 12 | 7 | 10 | 10 | 5 | 9 | 19 | 10 | 14 | 7 | 10 | 10 | 14 | 11 |
| New Dale-Chall | 9 | 7 | 11 | 9 | 7 | 11 | 14 | 10 | 11 | 9 | 10 | 10 | 11 | 10 |
The texts vary in terms of number of sentences (2–6 sentences), word count (13–89 words), and average words per sentence (5.3–17.8). This variation is expected since these are authentic texts included in the same brochure. For this experiment, we considered it more important to use texts that our participants could plausibly come across when looking for official information on vaccination than to expose them to texts of a similar length.
The texts were also compared in terms of readability (see Table 4). To this end, and similarly to reception studies in audiovisual translation (e.g., Van Hoecke, Schrijver, and Robert 2022Van Hoecke, Senne M., Iris Schrijver, and Isabelle S. Robert 2022 “Methodological Preparation of a Within-Subject Audiovisual Cognition, Reception and Perception Study.” Journal of Audiovisual Translation 5 (1): 94–128. ), we used three readability tests: the Flesch Reading Ease, the Flesch-Kincaid Grade Level, and the New Dale-Chall.
Both the Flesch Reading Ease and the Flesch-Kincaid Grade Level use the number of syllables and sentence lengths to calculate the reading ease of a text. In the case of the Flesch Reading Ease, a score below fifty indicates that the text is difficult to read, and below thirty corresponds to college level. The Flesch-Kincaid Grade Level indicates the number of years of education needed to understand a text using the American grade school level. Rather than relying on the length of words and sentences, the New Dale-Chall formula is calculated based on a list of 3000 words that fourth-grade American students (usually 9 to 10 years old) are expected to understand, calculating the readability score based on how many words in the text are not in this list. A score of five or lower indicates the passage is easily readable by the average fourth grader. Scores of nine or higher indicate that the text is at a college level of readability
As a reference, the US Department of Health and Human Services, the American Medical Association, and the National Institutes of Health recommend that online health information be written at or below a sixth-grade reading level (typically 11 to 12 years old) (ODPHP 2016ODPHP 2016 Health Literacy Online: A Guide for Simplifying the User Experience. 2nd ed. health.gov/healthliteracyonline). The Patient Information Forum in the UK (2023Patient Information Forum 2023 Health and Digital Literacy Survey 2022/23. London: Patient Information Forum. https://pifonline.org.uk/resources/publications/health-and-digital-literacy-survey-2223/) recommends that health information target a reading age of nine to eleven.
As shown in Table 4, all texts, except two, are estimated to be difficult to read. This aligns with previous research indicating that vaccine information materials may be too difficult for the average native speaker (Okuhara et al. 2022Okuhara, Tsuyoshi, Hirono Ishikawa, Haruka Ueno, Hiroko Okada, Mio Kato, and Takahiro Kiuchi 2022 “Readability Assessment of Vaccine Information: A Systematic Review for Addressing Vaccine Hesitancy.” Patient Education and Counseling 105 (2): 331–338. ). We assume that the readability levels for non-native speakers should be lower than those for native speakers. Therefore, we hypothesize that the Spanish group will experience difficulties understanding these texts — except for Text 2 and 5.
3.5Data analysis
We gauged readers’ self-reported difficulty using the following subjective measures:
-
Stops-making-sense judgments (online method) — operationalized by the number of times participants pressed the button ‘f’ per text during the reading task, which participants were instructed to do every time they encountered a region that did not make sense to them in the context of the text or that they did not understand.
-
‘I don’t know’ responses (offline method) — operationalized by the number of times participants selected the ‘I don’t know’ option when answering a comprehension question in the post-hoc comprehension questionnaire, which participants were instructed to select when they were not sure of the correct answer.
-
Perceived level of difficulty (offline method) — operationalized by participants’ rating of texts’ difficulty and the answer to the open question “What did you find most difficult to understand in these texts?” from the post-hoc comprehension questionnaire.
We gauged readers’ observed difficulty using the following objective measures:
-
Comprehension performance outcome (offline method) — operationalized by the number of times participants answered correctly and incorrectly to a comprehension question per text in the post-hoc comprehension questionnaire.
-
Raw reading time (online method) – raw reading time refers to the total amount of time a participant spent reading a text without any adjustments or processing.
-
Residual reading time (ResRT) (online method) — ResRTs per word are calculated to account for individual differences and varying word length. The ResRTs are calculated based on a linear fit (i.e., linear regression model) of the reading time of every word collected in the experiment against each word’s length. This fit is performed for every participant to correct for individual variations in the reading speed. The ResRTs are therefore obtained by subtracting for every word the predicted reading time based on the model fit for a given participant from the observed (raw) reading times (Ferreira and Clifton 1986Ferreira, Fernanda, and Charles Clifton 1986 “The Independence of Syntactic Processing.” Journal of Memory and Language 25 (3): 348–368. ). Linear regression was performed using the lm() function from the stats package in R (R Core Team 2021R Core Team 2021 “R: A Language and Environment for Statistical Computing.” R Foundation for Statistical Computing. https://www.R-project.org/).
To assess the statistical significance of the observed ResRT differences, we fitted a generalized linear mixed effects model using the “lme4” package (Bates et al. 2015Bates, Douglas, Martin Mächler, Ben Bolker, and Steve Walker 2015 “Fitting Linear Mixed-Effects Models Using Lme4.” Journal of Statistical Software 67 (1): 1–48. ) in R to predict the log-transformed and shifted ResRT data. Reading time data is commonly heavily skewed and requires transformation for linear regression analysis to meet the required residuals normality and homoscedastic conditions. This can be achieved by a data transformation or by applying a transformation link function in the General Linear Mixed Models (Lo and Andrews 2015Lo, Steson, and Sally Andrews 2015 “To Transform or Not to Transform: Using Generalized Linear Mixed Models to Analyse Reaction Time Data.” Frontiers in Psychology 6. ). To apply a logarithm transformation for the ResRTs the values have to be positive, so an offset of two seconds was applied to the distribution, resulting in the following transformation: TransformedResRT = log10(ResRT + 2). The model included Participant group and Text as predictors for main effects as well as an interaction term and participant as random effects. The best-fitting model was identified by reduction from a saturated model with all fixed effects and random effects included. The final model retained the main effect of Text, an interaction of Text and Participant group and a random structure with intercept by participant (formula: logResRT ~ 1 + Text + Text: Participant Group + (1|Participant)). Marginal means and contrasts for each of the texts between the English and Spanish groups were calculated using the “easystats” package (Lüdecke et al. 2022Lüdecke, Daniel, Indrajeet Patil, Mattan S. Ben-Shachar, Brenton M. Wiernik, Etienne Bacher, Rémi Thériault, and Dominique Makowski 2022 “easystats: Framework for Easy Statistical Modeling, Visualization, and Reporting.” CRAN. https://easystats.github.io/easystats/).
The data from both questionnaires collected in Qualtrics were exported to Excel, where we cleaned and analyzed the quantitative data from the close-ended questions. The responses to the open-ended question were analyzed using Atlas.ti 23. Here, the data were coded and organized around recurring themes (thematic analysis) using inductive coding drawing on Braun and Clarke (2006)Braun, Virginia, and Victoria Clarke 2006 “Using Thematic Analysis in Psychology.” Qualitative Research in Psychology 3 (2): 77–101. . Participants’ responses are quoted verbatim.
4.Results
The findings from readers’ self-reported difficulty and observed difficulty are presented below.
4.1Results on readers’ self-reported reading difficulty
4.1.1Stops-making-sense judgments
Overall, participants reported that the text stopped making sense ninety-six times (1.55% of 6182 potential regions). As expected, the Spanish group reported a higher number of times that the texts stopped making sense compared to the English group. The Spanish group reported seventy-three times (2.36% of 3091 potential regions), while the English group reported twenty-three (0.74%). Even though the reported frequency of stops-making-sense judgments was low, it is still surprising that the English group reported any instances at all, given that these target texts were presumably translated with this readership in mind.
When looking at stops-making-sense judgments per text (see Figure 3), the texts with the highest number of reported instances (i.e., 10 or more times) were Text 4, 7, 10, and 13, with Text 4 having the highest number of reported instances. In most cases, the Spanish group reported a higher number of instances of the text not making sense. The texts that did not present any issues — at least, considering only the stops-making-sense judgments — were Text 2 and 6. Most of the reported stops-making-sense instances correspond to Latin-Greek terms such as pertussis in Text 4 or epiglottitis in Text 7 (see as an example the stops-making-sense regions of Text 7 in Table 5).
| English group | Spanish group | Total | |
|---|---|---|---|
| Hib disease: | 1 | 3 | 4 |
| The Hib bacterium | 1 | 2 | 3 |
| causes | 0 | 0 | 0 |
| serious infections | 0 | 0 | 0 |
| such as blood poisoning | 0 | 0 | 0 |
| (septicaemia) | 1 | 1 | 2 |
| meningitis | 0 | 0 | 0 |
| epiglottitis | 2 | 3 | 5 |
| pneumonia | 0 | 0 | 0 |
| or inflammatory arthritis | 0 | 0 | 0 |
4.1.2‘I don’t know’ responses
Across both groups, the participants selected the ‘I don’t know’ option twenty-two times (3.45%) when asked to answer a comprehension question. In parallel to the stops-making-sense judgments, the Spanish group demonstrated a higher prevalence of selecting the ‘I don’t know’ option than the English group: fifteen times (4.70%) versus seven (2.19%).
When zooming in on the ‘I don’t know’ responses per text (see Figure 4), the texts for which the highest number of ‘I don’t know’ responses were recorded (i.e., 3 or more times) were Text 4, 7, 8, and 10. This corresponds in part to the results from the stops-making-sense judgment task: in both cases, the data suggests that Text 4, 7, and 10 were (more) difficult to understand. For Text 1, 2, 3, 5, and 12, no ‘I don’t know’ responses were registered.
4.1.3Perceived level of difficulty
Most participants from both groups reported that they found the texts easy to read, as indicated by their selection of the response options ‘Somewhat easy’ or ‘Extremely easy’ (see Figure 5).
In response to the question, “What did you find most difficult to understand in these texts?” two primary themes emerged from the data: difficulty understanding medical terminology (with 17 mentions) and difficulty understanding the comprehension questions (with 5 mentions) (see Figure 6).
The most commonly reported difficulty by both groups was related to medical terminology, which aligns with the findings from the stops-making-sense judgments. Participants indicated that they were unfamiliar with some of the medical terms in the texts (9 mentions). For example, one of the participants in the Spanish group wrote: “Some names of diseases, parts of the body in some cases or symptom [sic] of diseases that are related to each other.” Other participants reported difficulty understanding or lacking knowledge of Latin-Greek terms (5 mentions). Two participants took this opportunity to express that the terms were not explained “in more layman terms,” as one participant put it. Commenting on this, participants referred to an absence of a definition or explanation, as the comment below from one of the participants from the English group illustrates:
Sometimes the text lists the illness leading to pneumonia, for example. I happen to know the definition of pneumonia, but there were other examples that were listed in one word, without further definition. (…) it could have been more helpful if some/all of those medical terms were further defined in parenthesis.
In another case, a participant, also from the English group, argued that what was most difficult was “some of the medical terms used to explain the other medical terms.”
Participants’ second most common reported difficulty was not that the texts were challenging to understand but rather the comprehension questions (5 mentions), including their formulation, or that the options provided did not match their interpretation. Most participants who found the questions difficult to understand were native English speakers (4 vs 1).
A smaller number of participants (2) mentioned other difficulties, such as the texts having “too much information,” which contributed to the difficulty of “keeping track with the list of symptoms that a disease can cause.” In another case, one participant from the English group commented: “I was not sure that all the texts were correct about the diseases they described,” which we interpreted as skepticism or a lack of trust towards the content of the texts.
4.2Results on readers’ observed difficulty
4.2.1Comprehension performance outcome
Participants responded to most questions correctly: on average, participants answered 1.7 comprehension questions incorrectly (out of 29). As anticipated, the Spanish group had a higher number of incorrect answers, with twenty incorrect answers, but their behavior is similar to the English group, totaling eighteen incorrect answers. The texts that prompted the most incorrect comprehension answers were Text 8 and 13. In relation to Text 6, 9, 10, 11, and 12, all the questions were answered correctly (see Figure 7).
Both groups behaved similarly when answering most questions, except about Text 13. In that case, the Spanish group made a significantly higher number of mistakes, all in response to the same question: “What are some of the permanent health-related problems for a child that had meningococcal disease?” And while the correct answer was “deafness; difficulties learning; amputation of a leg or arm,” six participants in the Spanish group selected only one or two of these problems. Even though participants could reread the text as many times as they wanted and the text was present on screen with the comprehension questions, it seems that the participants, particularly the Spanish group, had difficulties retrieving the information. This aligns with two answers to the question “What did you find most difficult to understand in these texts?” where Spanish participants identified information overload as one of their main difficulties.
4.2.2Raw reading time
Our initial hypothesis postulated that the Spanish group would exhibit longer reading times across all target texts. Normal reading time values of 250–400 ms per word for English (Brysbaert 2019Brysbaert, Marc 2019 “How Many Words Do We Read per Minute? A Review and Meta-Analysis of Reading Rate.” Journal of Memory and Language 109: 104047. ) and 350–550 ms per word for Spanish (Park 2022Park, Jeongyeon 2022 “Promoting L2 Reading Fluency at the Tertiary Level Through Timed and Repeated Reading.” System 107: 102802. ) were considered as thresholds to determine difficulty in reading.
Overall, both groups took longer than expected to read most texts: in the case of the English group, only three texts were read at a speed below 400 ms per word, and in the case of the Spanish group, only five texts were read at a speed below 550 ms per word. Nevertheless, these observations can be affected by individual participants’ performance differences and variability in words’ length. To account for this, statistical analysis was performed on the calculated log-transformed ResRT.
4.2.3ResRT
Our initial hypothesis postulated that the native Spanish speakers would exhibit longer ResRTs across all target texts. We thus expected to find an effect of group on reading times. Based on readability metrics, we also anticipated that Text 2 and 5 would demand shorter ResRTs (see Section 3.3). Upon examining the ResRT data, it is evident that the English group shows, in general, larger ResRTs than the native Spanish speakers (see Figure 8). Descriptively speaking, Text 7 emerged as the text eliciting the highest ResRT across both groups, while Text 2 showed the lowest.
A closer examination of the statistical analysis of the log-transformed ResRT data derived from each text for both participant groups highlights a statistically significant disparity between the English and Spanish groups (see Table 6). Specifically, the English group exhibited statistically significantly higher ResRTs across a greater number of texts. Text 1, 3, 5, 11, 13, and 14 showed a significantly higher ResRT in the English group, compared with the Spanish group. Conversely, Text 4, 10, and 12 had significantly higher ResRTs for the Spanish group compared to the native English speakers. These results do not confirm our expectations, given that the English group showed significantly higher ResRTs across a greater number of texts in comparison to the Spanish group.
| Level1 | Level2 | Target text | Difference | SE | df | t-value | p-value | 95% CI |
|---|---|---|---|---|---|---|---|---|
| English | Spanish | 1 | 0.03 | 0.01 | 1827.30 | 2.37 | .018* | [0.01, 0.06] |
| English | Spanish | 2 | −0.00 | 0.02 | 2506.73 | −0.21 | .831 | [−0.03, 0.03] |
| English | Spanish | 3 | 0.04 | 0.02 | 4076.09 | 2.08 | .038* | [0.00, 0.08] |
| English | Spanish | 4 | −0.06 | 0.01 | 1205.12 | −5.46 | < .001*** | [−0.09, −0.04] |
| English | Spanish | 5 | 0.06 | 0.01 | 1836.98 | 4.16 | < .001*** | [0.03, 0.08] |
| English | Spanish | 6 | 0.03 | 0.02 | 4498.01 | 1.31 | .192 | [−0.01, 0.07] |
| English | Spanish | 7 | 0.02 | 0.02 | 4217.58 | 1.08 | .281 | [−0.02, 0.06] |
| English | Spanish | 8 | −0.04 | 0.02 | 3880.64 | −1.96 | .050 | [−0.08, 0.00] |
| English | Spanish | 9 | 0.03 | 0.02 | 2443.41 | 1.77 | .077 | [−0.00, 0.06] |
| English | Spanish | 10 | −0.05 | 0.02 | 3078.62 | −2.75 | .006** | [−0.08, −0.01] |
| English | Spanish | 11 | 0.03 | 0.01 | 2,281.18 | 2.15 | .032* | [0.00, 0.06] |
| English | Spanish | 12 | −0.03 | 0.01 | 1,418.58 | −2.06 | .040* | [−0.05, −0.00] |
| English | Spanish | 13 | 0.03 | 0.01 | 1,035.71 | 2.33 | .020* | [0.00, 0.05] |
| English | Spanish | 14 | 0.02 | 0.01 | 1,306.39 | 1.36 | .173 | [−0.01, 0.04] |
SE = standard error, CI = confidence interval, df = degrees of freedom;
5.Conclusion
This article reported on a quasi-experiment investigating whether lay readers experienced reading difficulty and misinterpreted translated vaccination-related information. To do so, it employed a participant-paced, region-by-region reading task combined with an incremental judgment task and a post-hoc comprehension questionnaire.
The target texts were official English translations of the Dutch brochure of the National Immunization Program, published by the Dutch Health Ministry under the heading “Translations of Letters and Brochures” on the RIVM website. The use of several readability measures allowed us to hypothesize that most of these texts were more suitable for college-level readers, which would, in principle, imply that they are rather inaccessible for non-native English speakers, such as the ones from our Spanish group. This also led us to hypothesize that these participants would misunderstand the information in these texts.
The results, based on the triangulation of multiple data sources, indicate that both groups of participants experienced some degree of reading difficulty and misinterpretation. The extent of the reading difficulty and misunderstanding varied by text and, in some cases, by group. Overall, data triangulation suggests that Text 7, 10, and 13 were the most difficult for both groups of participants, and Text 2 was the easiest.
Both the English and the Spanish groups behaved similarly regarding the total number of errors in response to comprehension questions. It would have been expected that the English group would exhibit few to no errors in comprehension or at least a marked difference in terms of fewer errors than the Spanish group. After all, it is safe to assume that native English speakers are the ideal readers of these target texts and that these ‘informed readers’ are ideally competent to understand and act on the target texts (see Fish 1970Fish, Stanley 1970 “Literature in the Reader: Affective Stylistics.” New Literary History 2 (1): 123–162. on ideal readers).77.This does not preclude that non-native speakers of English are considered implied readers and that the translators did have both non-native speakers of English as well as native English speakers in mind when producing the target texts. However, this was not the case. Even if participants could reread the texts as many times as they wanted when answering comprehension questions, and no memory or time limitations were involved, participants from both groups made comprehension errors.
What was also unexpected was that native English speakers exhibited statistically significant higher ResRTs for a broader range of texts when compared to native Spanish speakers. This suggests that there are specificities in the target texts that make them longer to read for the English group. We posit that those specificities are medical terms of Latin-Greek origin, which might be more familiar to Spanish speakers (see Jiménez-Crespo and Tercedor Sánchez 2017Jiménez-Crespo, Miguel A., and Maribel Tercedor Sánchez 2017 “Lexical Variation, Register and Explicitation in Medical Translation.” Translation and Interpreting Studies 12 (3): 405–426. ). However, more research on this topic needs to be undertaken before this association is more clearly understood.
Our results also point to what was most difficult to understand in the target texts. These insights provide useful information for translators tasked with translating healthcare information for non-experts. Based on these observations, we propose actionable recommendations for translators to consider.
Data from participants’ stops-making-sense judgments, ‘I don’t know’ responses, comprehension questions, and perceived difficulty levels suggest that medical language, particularly terminology, poses hurdles for lay readers. This aligns with prior research on medical translation, which identified medical terminology, particularly Latin-Greek terms, as a common source of issues (see Section 2.2).
Drawing upon participant feedback, we recommend that translators provide clear and concise explanations and definitions of medical terms using plain, non-technical language. Avoiding the use of medical terms to explain other medical terms might help to prevent additional comprehension difficulties.
The medical terms in the target texts, described by some participants as “unfamiliar,” refer to diseases covered by vaccination programs at a global level. This unfamiliarity with diseases that are covered globally suggests low health literacy. Even if it is not within the scope of this article to discuss health literacy, health literacy strategies involve avoiding technical jargon in materials aimed at lay readers and providing translated information (e.g., Patient Information Forum 2023Patient Information Forum 2023 Health and Digital Literacy Survey 2022/23. London: Patient Information Forum. https://pifonline.org.uk/resources/publications/health-and-digital-literacy-survey-2223/). This is particularly significant when research indicates that difficulty reading vaccine information may influence vaccine hesitancy (Okuhara et al. 2022Okuhara, Tsuyoshi, Hirono Ishikawa, Haruka Ueno, Hiroko Okada, Mio Kato, and Takahiro Kiuchi 2022 “Readability Assessment of Vaccine Information: A Systematic Review for Addressing Vaccine Hesitancy.” Patient Education and Counseling 105 (2): 331–338. ).
Furthermore, our findings suggest that information overload can also contribute to misinterpretations. In other words, lay readers may struggle to process and retain information presented in lengthy enumerations. It is thus important to consider shortening texts and breaking down longer sentences into smaller, more manageable units.
It is evident that the Dutch Ministry has taken its duty to communicate with the broader and linguistically diverse communities in the Netherlands seriously by providing translations of vaccination-related public health information into English. Our results do not invalidate this effort. However, it is important to note that providing access to translated information is not enough to ensure comprehension. And as the findings in this study suggest, we should not underestimate the complexity of reading and comprehending translated medical texts. Future studies should, therefore, continue to investigate how migrants read public health information, particularly the relation between readers’ cognitive effort and different translation strategies and how it affects comprehension. This is the aim of our follow-up experiment, which will be based on the reading data collected, as well as the lessons learned from designing and conducting this study.
An important limitation of the study is related to the number and profile of the participants, which could have influenced the results. The high level of proficiency in English and the level of education of the participants in the Spanish group may be relevant factors. It is also relevant to acknowledge that some participants considered the comprehension questions difficult, and, therefore, it is recommended that future studies also control for this potential limitation.
Funding
Open Access publication of this article was funded through a Transformative Agreement with Leiden University.
Acknowledgements
We would like to thank all readers who contributed to this project.