Cover not available

Article published In: Questionable Research Practices in Applied Linguistics
Edited by Luke Plonsky
[Journal of Second Language Studies 8:2] 2025
► pp. 192–218

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

The Ulysses pact

An emic perspective on registered reports in applied linguistics

 | Beijing Foreign Studies University, China
 | Shanghai Jiao Tong University, China
 | Florida State University, USA
 | Royal Commission for Jubail and Yanbu, Saudi Arabia
Published online: 14 April 2025
DOI logo with link
https://doi.org/10.1075/jsls.00045.liu
Abstract
Registered reports (RRs) are gaining traction in applied linguistics as a means to enhance research transparency and credibility by disincentivizing questionable research practices that are aimed at generating statistically significant findings, and by mitigating publication bias. While the benefits of RRs are well-theorized in the literature, less is known about authors’ experiences of conducting RRs. This study explored the first-hand experiences of 12 authors of RRs in applied linguistics from an emic perspective currently underrepresented in the field. Through semi-structured interviews, we examined authors’ motivations for engaging in RRs, perceived benefits and challenges, and reflections on RRs. Our findings revealed that authors valued RRs for promoting scientific rigor and offering publication guarantee and found the process to be highly beneficial. However, tensions were also found in the process, ranging from the potentially time-consuming nature of RRs to reduced autonomy and role ambiguity experienced by some authors. Based on these insights, we offer recommendations for improving the RR process and call for greater support for all stakeholders in the process, including reviewers and editors. We also compiled a list of recommendations by our participants to aid future authors in choosing and navigating RRs.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Overview of RRs
      • 2.1.1The RR workflow
      • 2.1.2Adoption of RRs
    • 2.2Benefits of RRs
      • 2.2.1Disincentivizing QRPs
      • 2.2.2Mitigating publication bias
      • 2.2.3Enhancing transparency and reproducibility
    • 2.3Challenges of RRs
    • 2.4Experience of conducting RRs
  • 3.The present study
  • 4.Method
    • 4.1Participants
    • 4.2Data collection
    • 4.3Data analysis
  • 5.Results
    • 5.1Incentives for conducting RRs
    • 5.2Benefits and positive experience of engaging in RRs
    • 5.3Challenges encountered in RRs
    • 5.4Advice for future researchers
  • 6.Discussion
    • 6.1Aligning publication incentives with quality standards
    • 6.2Tensions in the process of RRs
    • 6.3Implications for future RRs and open science initiatives
    • 6.4Recommendations for future authors
    • 6.5Limitations and future directions
  • 7.Conclusion
  • Note
  • References
References (37)
References
Al-Hoorie, A. H., Cinaglia, C., Hiver, P., Huensch, A., Isbell, D. R., Leung, C., & Sudina, E. (2024). Open science: Considerations and issues for TESOL research. TESOL Quarterly. Google Scholar logo with link to Google Scholar
Allen, C., & Mehler, D. M. A. (2019). Open science challenges, benefits and tips in early career and beyond. PLOS Biology, 17(5), e3000246. Google Scholar logo with link to Google Scholar
Benson, P. (2019). Narrative analysis. In A. Phakiti, P. De Costa, L. Plonsky, & S. Starfield (Eds.), The Palgrave handbook of applied linguistics research methodology (pp. 595–613). Springer.Google Scholar logo with link to Google Scholar
Bolibaugh, C., Vanek, N., & Marsden, E. (2021). Towards a credibility revolution in bilingualism research: Open data and materials as stepping stones to more reproducible and replicable research. Bilingualism: Language and Cognition, 24(5), 801–806. Google Scholar logo with link to Google Scholar
Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. Google Scholar logo with link to Google Scholar
Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29–42. Google Scholar logo with link to Google Scholar
Czarniawska, B. (2004). Narratives in social science research. Sage. Google Scholar logo with link to Google Scholar
DeSantis, L., & Ugarriza, D. N. (2000). The concept of theme as used in qualitative nursing research. Western Journal of Nursing Research, 22(3), 351–372. Google Scholar logo with link to Google Scholar
Drax, K., Clark, R., Chambers, C. D., Munafò, M., & Thompson, J. (2021). A qualitative analysis of stakeholder experiences with registered reports funding partnerships. Wellcome Open Research, 61, 230. Google Scholar logo with link to Google Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One, 4(5), e5738. Google Scholar logo with link to Google Scholar
Godfroid, A., & Andringa, S. (2023). Uncovering Sampling Biases, Advancing Inclusivity, and Rethinking Theoretical Accounts in Second Language Acquisition: Introduction to the Special Issue SLA for All? Language Learning, lang.12620. Google Scholar logo with link to Google Scholar
Godlee, F. (2002). Making reviewers visible: Openness, accountability, and credit. JAMA, 287(21), 2762–2765. Google Scholar logo with link to Google Scholar
Hardwicke, T. E., & Ioannidis, J. P. A. (2018). Mapping the universe of registered reports. Nature Human Behaviour, 2(11), 793–796. Google Scholar logo with link to Google Scholar
Huensch, A. (2024). Open science and preregistration. In L. Plonsky (Ed.), Open science in applied linguistics (pp. 1–22). John Benjamins.Google Scholar logo with link to Google Scholar
Hummer, L., Thorn, F. S., Nosek, B. A., & Errington, T. M. (2017). Evaluating registered reports: A naturalistic comparative study of article impact. [URL]
Isbell, D. R., Brown, D., Chen, M., Derrick, D. J., Ghanem, R., Arvizu, M. N. G., … Plonsky, L. (2022). Misconduct and questionable research practices: The ethics of quantitative data handling and reporting in applied linguistics. Modern Language Journal, 106(1), 172–195. Google Scholar logo with link to Google Scholar
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. Google Scholar logo with link to Google Scholar
Lakens, D. (2019). The value of preregistration for psychological science: A conceptual analysis. Japanese Psychological Review, 62(3), 221–230. Google Scholar logo with link to Google Scholar
Larsson, T., Plonsky, L., Sterling, S., Kytö, M., Yaw, K., & Wood, M. (2023). On the frequency, prevalence, and perceived severity of questionable research practices. Research Methods in Applied Linguistics, 2(3), 100064. Google Scholar logo with link to Google Scholar
Laurinavichyute, A., Yadav, H., & Vasishth, S. (2022). Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy. Journal of Memory and Language, 1251, 104332. Google Scholar logo with link to Google Scholar
Liu, M., Al-Hoorie, A. H., & Hiver, P. (2024). Open access in language testing and assessment: The case of two flagship journals. Language Testing. Google Scholar logo with link to Google Scholar
Liu, M., & Marsden, E. (2024). The open turn: Rethinking applied linguistics research through open scholarship. OSF. Google Scholar logo with link to Google Scholar
Marsden, E., Crossley, S., Ellis, N., Kormos, J., Morgan-Short, K., & Thierry, G. (2019). Inclusion of research materials when submitting an article to Language Learning. Language Learning, 69(4), 795–801. Google Scholar logo with link to Google Scholar
Marsden, E., & Morgan-Short, K. (2023). (Why) are open research practices the future for the study of language learning? Language Learning, 1–44. Google Scholar logo with link to Google Scholar
Marsden, E., Morgan-Short, K., Trofimovich, P., & Ellis, N. C. (2018). Introducing registered reports at language learning: Promoting transparency, replication, and a synthetic ethic in the language sciences. Language Learning, 68(2), 309–320. Google Scholar logo with link to Google Scholar
Montoya, A. K., Krenzer, W. L. D., & Fossum, J. L. (2021). Opening the door to registered reports: Census of journals publishing registered reports (2013–2020). Collabra: Psychology, 7(1), 24404. Google Scholar logo with link to Google Scholar
Munafò, M. R. (2017). Improving the efficiency of grant and journal peer review: Registered reports funding. Nicotine & Tobacco Research, 19(7), 773–773. Google Scholar logo with link to Google Scholar
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. Google Scholar logo with link to Google Scholar
Obels, P., Lakens, D., Coles, N. A., Gottfried, J., & Green, S. A. (2020). Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science, 3(2), 229–237. Google Scholar logo with link to Google Scholar
Parsons, S., Azevedo, F., Elsherif, M. M., Guay, S., Shahim, O. N., Govaart, G. H., … Todorovic, A. (2022). A community-sourced glossary of open scholarship terms. Nature Human Behaviour, 6(3), 312–318. Google Scholar logo with link to Google Scholar
Plonsky, L. (2024). Open science in applied linguistics. Applied Linguistics Press.Google Scholar logo with link to Google Scholar
Plonsky, L., Brown, D., Chen, M., Ghanem, R., Arvizu, M. N. G., Isbell, D. R., & Zhang, M. (2024). “significance sells”: Applied linguists’ views on questionable research practices. Research Methods in Applied Linguistics, 3(1), 100099. Google Scholar logo with link to Google Scholar
Saldaña, J. (2016). The coding manual for qualitative researchers. Sage.Google Scholar logo with link to Google Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. Google Scholar logo with link to Google Scholar
Steneck, N. H. (2006). Fostering integrity in research: Definitions, current knowledge, and future directions. Science and Engineering Ethics, 121, 53–74. Google Scholar logo with link to Google Scholar
Waltman, L., Kaltenbrunner, W., Pinfield, S., & Woods, H. B. (2023). How to improve scientific peer review: Four schools of thought. Learned Publishing, 36(3), 334–347. Google Scholar logo with link to Google Scholar
Wood, M., Sterling, S., Larsson, T., Plonsky, L., Kytö, M., & Yaw, K. (2024). Researchers training researchers: Ethics training in quantitative applied linguistics. TESOL Quarterly, tesq.3323. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue