Cover not available

Article published In: Questionable Research Practices in Applied Linguistics
Edited by Luke Plonsky
[Journal of Second Language Studies 8:2] 2025
► pp. 344–374

Get fulltext from our e-platform
JBE logo with link
EPUB logo with download linkDownload EPUB
authorization required

Exploring questionable research practices in applied linguistics mixed methods research studies

ORCiD logo with link | Iran University of Science & Technology, Iran
ORCiD logo with link | Hamad Bin Khalifa University, Qatar
Published online: 12 September 2025
DOI logo with link
https://doi.org/10.1075/jsls.00051.far
Abstract
Beginning in 2022, the field of applied linguistics has increasingly approached the evaluation of research quality through an ethical lens, with a particular emphasis on Questionable Research Practices (QRPs). Notably, the majority of existing investigations into QRPs have concentrated on mono-method studies, especially those employing quantitative methodologies, thereby neglecting the realm of mixed methods research (MMR). The present study seeks to illuminate the problematic areas that may contribute to QRPs within MMR studies. To this end, we analyzed 60 MMR studies published between 2011 and 2020 in leading journals within the domain of applied linguistics (AL). Our findings reveal a range of issues pertaining to MMR rhetoric and references, study purpose and design, as well as the integration of methodologies, all of which pose risks to the transparency and foundational principles of MMR. This study concludes with recommendations aimed at enhancing the quality of MMR studies.
Article outline
  • 1.Introduction
    • 1.1Quality Culture in AL
    • 1.2Related Studies: QRPs in Applied Linguistics
  • Methods
    • 2.1MMR studies identification and retrieval
    • 2.2Analysis of the 60 articles concerning QPRs
  • 3.Results
    • 3.1MMR Language and MMR References
      • 3.1.1MMR Language and citing relevant sources
      • 3.1.2MMR references
    • 3.2MMR purpose
    • 3.3MMR Design
    • 3.4Integration
  • 4.Discussion
  • 5.Conclusions
  • References
References (79)
References
Al-Hoorie, A. H., & Vitta, J. P. (2019). The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore, SJR, SNIP, JCR Impact Factors. Language Teaching Research, 23(6), 727–744. Google Scholar logo with link to Google Scholar
Amini Farsani, M. (2025). Enriching quantitative phase in mixed methods research. In The Encyclopedia of Applied Linguistics (Mixed Methods), C. A. Chapelle & M. Riazi (Eds.). Google Scholar logo with link to Google Scholar
Amini Farsani, M., Jamali, H. R., Beikmohammadi, M., Daneshvar Ghorbani, B., & Soleimani, L. (2021). Methodological orientations, academic citations, and scientific collaboration in applied linguistics: What do research synthesis and bibliometrics indicate? System, 1001, 102547. Google Scholar logo with link to Google Scholar
Amini Farsani, M., Babaii, E., Beikmohammadi, M., & Babaii Farsani, M. (2022). Mixed-methods research proficiency for applied linguists: A PLS-path modelling approach. Quality & Quantity, 56(5), 3337–3362. Google Scholar logo with link to Google Scholar
Alise, M., & Teddlie, C. (2010). A continuation of the paradigm wars? Prevalence rates of methodological approaches across the social/behavioral sciences. Journal of Mixed Methods Research, 4(2), 103–126. Google Scholar logo with link to Google Scholar
Anderson, C. (2017). Ethics in Qualitative Language Education Research. In: Mirhosseini, S. A. (eds) Reflections on Qualitative Research in Language and Literacy Education. Educational Linguistics, vol 291. Springer, Cham. Google Scholar logo with link to Google Scholar
Brown, J. D. (2009). Open-response items in questionnaires. In Heigham, J. and Croker, R. A. (eds.), Qualitative research in applied linguistics: A practical introduction. Basingstoke, UK: Palgrave Macmillan, 200–219. Google Scholar logo with link to Google Scholar
Chapelle, C. (2012). Using mixed-methods research in technology-based innovation for language learning. Paper presented at the Innovative Practices in Computer Assisted Language Learning Conference, University of Ottawa, Ontario.
Chen, Q., & Wright, C. (2017). Contextualization and authenticity in TBLT: Voices from Chinese classrooms. Language Teaching Research, 21(4), 517–538. Google Scholar logo with link to Google Scholar
Chong, S. W., & Plonsky, L. (2024). Research synthesis in language education. In International Encyclopedia of Language and Linguistics (3rd ed.). Elsevier. Google Scholar logo with link to Google Scholar
Creamer, E. G. (2018). An introduction to fully integrated mixed methods research. Sage. Google Scholar logo with link to Google Scholar
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and conducting mixed methods research (2nd ed.). Sage.Google Scholar logo with link to Google Scholar
(2018). Designing and conducting mixed methods research (3rd ed.). Sage.Google Scholar logo with link to Google Scholar
De Costa, P. I. (Ed.). (2015). Ethics in applied linguistics research: Language researcher narratives. Routledge. Google Scholar logo with link to Google Scholar
Dolgova Jacobsen, N. (2018). The best of both worlds: Combining cognitive linguistics and pedagogic tasks to teach English conditionals. Applied Linguistics, 39(5), 668–693. Google Scholar logo with link to Google Scholar
Dörnyei, Z. (2007). Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford.Google Scholar logo with link to Google Scholar
Dufon, M. (1993). Ethics in TESOL research. TESOL Quarterly, 27(1), 157–160. Google Scholar logo with link to Google Scholar
F`abregues, S., & Molina-Azorin, J. F. (2017). Addressing quality in mixed methods research: A review and recommendations for a future agenda. Quality and Quantity, 51(6), 2847–2863. Google Scholar logo with link to Google Scholar
Fàbregues, S., Molina-Azorin, J. F., & Fetters, M. D. (2021). Virtual special issue on “quality in mixed methods research”. Journal of Mixed Methods Research, 15(2), 146–151. Google Scholar logo with link to Google Scholar
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS One, 4(5), e5738. Google Scholar logo with link to Google Scholar
Fetters, M. D., & Molina-Azorin, J. F. (2017). The journal of mixed methods research starts a new decade: The mixed methods research integration trilogy and its dimensions. Journal of Mixed Methods Research, 11(3), 291–307. Google Scholar logo with link to Google Scholar
Fetters, M. D., Curry, L. A., & Creswell, J. W. (2013). Achieving integration in mixed methods designs-principles and practices. Health Services Research, 48(6), 2134–2156. Google Scholar logo with link to Google Scholar
Gass, S., Loewen, S., & Plonsky, L. (2021). Coming of age: The past, present, and future of quantitative SLA research. Language Teaching, 54(2), 245–258. Google Scholar logo with link to Google Scholar
Ghanbar, H. & Rezvani, R. (forthcoming). Quality issues in applied Linguistics qualitative research. Journal of Modern Research in English Language Studies.
Gonulal, T., Loewen, S., & Plonsky, L. (2017). The development of statistical literacy in applied linguistics graduate students. ITL-International Journal of Applied Linguistics, 168(1), 4–32. Google Scholar logo with link to Google Scholar
Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis, 11(3), 255–274. Google Scholar logo with link to Google Scholar
Guetterman, T. C. (2015). The development, design, and test of a self-assessmnet instrument of mixed methods research proficiency [Unpublished Ph.D. Dissertation]. University of Nebraska.
(2017). What distinguishes a novice from an expert mixed methods researcher? Quality and Quantity, 51(1): 377–398. Google Scholar logo with link to Google Scholar
Guetterman, T. C., Plano Clark, V. L., & Molina-Azorin, J. F. (2024). Terminology and Mixed Methods Research: A Persistent Challenge. Journal of Mixed Methods Research, 18(1), 9–13. Google Scholar logo with link to Google Scholar
Hall, J., & Martin, B. R. (2019). Towards a taxonomy of research misconduct: The case of business school research. Research Policy, 48(2), 414–427. Google Scholar logo with link to Google Scholar
Harrison, R. L., Reilly, T. M., & Creswell, J. W. (2020). Methodological rigor in mixed methods: An application in management studies. Journal of Mixed Methods Research, 14(4), 473–495. Google Scholar logo with link to Google Scholar
Heyvaert, M., Maes, B., & Onghena, P. (2013). Mixed methods research synthesis: Definition framework, and potential. Quality and Quantity, 471, 659–676. Google Scholar logo with link to Google Scholar
Hirose, M., & Creswell, J. W. (2022). Applying core quality criteria of mixed methods research to an empirical study. Journal of Mixed Methods Research, 17(1), 12–28. Google Scholar logo with link to Google Scholar
Hudson, T., & Llosa, L. (2015). Design issues and inference in experimental L2 research. Language Learning, 65(S1), 76–96. Google Scholar logo with link to Google Scholar
Isbell, D., Brown, D., Chan, M., Derrick, D., Ghanem, R., Gutiérrez Arvizu, M. N., Schnur, E., Zhang, M., & Plonsky, L. (2022). Misconduct and questionable research practices: The ethics of quantitative data handling and reporting in applied linguistics. Modern Language Journal, 106(1), 172–195. Google Scholar logo with link to Google Scholar
Johnson, R. B. (2017). Dialectical pluralism: A metaparadigm whose time has come. Journal of Mixed Methods Research, 111, 156–173. Google Scholar logo with link to Google Scholar
Johnson, R. B., Onwuegbuzie, A. J., & Turner, L. A. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133. Google Scholar logo with link to Google Scholar
Kao, T. N. (2012). Factor analysis of English writing demotivation among central Taiwan university students (Unpublished master thesis). Taichung, Taiwan: Providence University.
King, K. A., & Mackey, A. (2016). Research methodology in second language studies: Trends, concerns, and new directions. The Modern Language Journal, 1001, 209–227. Google Scholar logo with link to Google Scholar
Kubanyiova, M. (2008). Rethinking research ethics in contemporary applied linguistics: The tension between macroethical and microethical perspectives in situated research. The Modern Language Journal, 92(4), 503–518. Google Scholar logo with link to Google Scholar
Larsson, T., Plonsky, L., Sterling, S., Kytö, M., Yaw, K., & Wood, M. (2023). On the frequency, prevalence, and perceived severity of questionable research practices. Research Methods in Applied Linguistics, 2(3), 100064. Google Scholar logo with link to Google Scholar
Lee, J. S., Nakamura, Y., & Sadler, R. (2018). Effects of videoconference-embedded classrooms (VEC) on learners’ perceptions toward English as an international language (EIL). ReCALL, 30(3), 319–336. Google Scholar logo with link to Google Scholar
Lei, L., & Liu, D. (2019). Research trends in applied linguistics from 2005 to 2016: A bibliometric analysis and its implications. Applied Linguistics, 401: 540–561. Google Scholar logo with link to Google Scholar
Marsden, E., Mackey, A., & Plonsky, L. (2016). Breadth and depth: The IRIS repository. In A. Mackey & E. Marsden (Eds.), Advancing methodology and practice: The IRIS repository of instruments for research into second languages (pp. 1–21). Routledge.Google Scholar logo with link to Google Scholar
Meihami, H., & Esfandiari, R. (Eds.). (2024). A Scientometrics Research Perspective in Applied Linguistics. Springer. Google Scholar logo with link to Google Scholar
Mertens, D. M. (2007). Transformative paradigm: Mixed methods and social justice. Journal of Mixed Methods Research, 11, 212–225. Google Scholar logo with link to Google Scholar
Moeller, A. J., & Theiler, J. (2014). Spoken Spanish language development at the high school level: A mixed-methods study. Foreign Language Annals, 47(2), 210–240. Google Scholar logo with link to Google Scholar
Ortega, L. (2005a). Methodology, epistemology, and ethics in instructed SLA research: An introduction. Modern Language Journal, 89(3), 317–327. Google Scholar logo with link to Google Scholar
(2005b). For what and for whom is our research? The ethical as transformative lens in instructed SLA. Modern Language Journal, 89(3), 427–443. Google Scholar logo with link to Google Scholar
Peace, M. M. (2015). Other-Orientation in Nonnative Spanish and Its Effect on Direct Objects. Foreign Language Annals, 48(4), 669–687. Google Scholar logo with link to Google Scholar
Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35(4), 655–687. Google Scholar logo with link to Google Scholar
(2024). Study quality as an intellectual and ethical imperative: A proposed framework. Annual Review of Applied Linguistics, 1–15. Google Scholar logo with link to Google Scholar
Plonsky, L., & Oswald, F. L. (2015). Meta-analyzing second language research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research (pp. 106–128). Routledge. Google Scholar logo with link to Google Scholar
Purpura, J. E., Brown, J. D., & Schoonen, R. (2015). Improving the validity of quantitative measures in applied linguistics research. Language Learning, 65(S1), 37–75. Google Scholar logo with link to Google Scholar
Riazi, A. M. (2016). Innovative mixed-methods research: Moving beyond design technicalities to epistemological and methodological realizations. Applied Linguistics, 37(1), 33–49. Google Scholar logo with link to Google Scholar
(2017). Mixed-methods research in language teaching and learning. Equinox.Google Scholar logo with link to Google Scholar
(2024). Mixed methods research (MMR): An overview. In The Encyclopedia of Applied Linguistics (Mixed Methods), C. A. Chapelle & M. Riazi (Eds.). Google Scholar logo with link to Google Scholar
Riazi, A. M., & Candlin, C. N. (2014). Mixed-methods research in language teaching and learning: Opportunities, issues, and challenges. Language Teaching, 47(2), 135–173. Google Scholar logo with link to Google Scholar
Riazi, A. M., & Amini Farsani, M. (2024). Mixed-methods research in applied linguistics: Charting the progress through the second decade of the twenty-first century. Language Teaching, 57(2), 143–182. Google Scholar logo with link to Google Scholar
Riazi, A. M., Farr, F., Han, Ch., & Amini Farsani, M. (2024). Fostering methodological awareness: The Mixed methods research affordances. Paper presented at the 21st AILA World Congress, Kuala Lumpur, Malysia
Robson, C. (2002). Real world research: A resource for social scientists and practitioner-researchers. John Wiley & Sons.Google Scholar logo with link to Google Scholar
Roy, D., Brine, J., & Murasawa, F. (2016). Usability of English note-taking applications in a foreign language learning context. Computer Assisted Language Learning, 29(1), 61–87. Google Scholar logo with link to Google Scholar
Schoonenboom, J. (2022). Developing the metainference in mixed methods research through successive integration of claims. In Hitchcock, J. H. and Onwuegbuzie, A. J. (eds.) The Routledge handbook for advancing integration in mixed methods research (pp. 55–70). Routledge. Google Scholar logo with link to Google Scholar
Steneck, N. H. (2007). Introduction to the responsible conduct of research. Department of Health and Human Services, Office of Research Integrity. [URL].
Sterling, S., & Gass, S. (2017). Exploring the boundaries of research ethics: Perceptions of ethics and ethical behaviors in applied linguistics research. System, 701, 50–62. Google Scholar logo with link to Google Scholar
Sterling, S., Winke, P., & Gass, S. (2016). Training in research ethics among applied linguistics and SLA researchers. In P. I. De Costa (Ed.), Ethics in applied linguistics research: Language researcher narratives (pp. 15–37). Routledge.Google Scholar logo with link to Google Scholar
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. Sage.Google Scholar logo with link to Google Scholar
(2009). Integrating qualitative and quantitative approaches to research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 283–318). Sage. Google Scholar logo with link to Google Scholar
Teddlie, C. B., and Tashakkori, A. (2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioural sciences. Sage.Google Scholar logo with link to Google Scholar
Thomas, M. (2009). Ethical issues in the study of second language acquisition: Resources for teachers. Second Language Research, 25(4), 493–511. Google Scholar logo with link to Google Scholar
Van Kemenade, E., Pupius, M., & Hardjono, T. W. (2008). More value to defining quality. Quality in Higher Education, 14(2), 175–185. Google Scholar logo with link to Google Scholar
Vitta, J. P., Hahn, A., Canning, D., Isbell, D. R., Al-Hoorie, A. H., & Nicklin, C. (2025). Conceptualization and frequency of sampling (mal)practices in L2 inferential quantitative research. Journal of Second Language Studies. [URL]
Vogel, S., Herron, C., Cole, S. P., & York, H. (2011). Effectiveness of a guided inductive versus a deductive approach on the learning of grammar in the intermediate-level college French classroom. Foreign language annals, 44(2), 353–380. Google Scholar logo with link to Google Scholar
Wette, R. (2017). Source text use by undergraduate post-novice L2 writers in disciplinary assignments: Progress and ongoing challenges. Journal of Second Language Writing, 371, 46–58. Google Scholar logo with link to Google Scholar
Wu, W. C. V., Yang, J. C., Scott Chen Hsieh, J., & Yamamoto, T. (2020). Free from demotivation in EFL writing: The use of online flipped writing instruction. Computer Assisted Language Learning, 33(4), 353–387. Google Scholar logo with link to Google Scholar
Yang, J. (2016). Learners’ oral corrective feedback preferences in relation to their cultural background, proficiency level and types of error. System, 611, 75–86. Google Scholar logo with link to Google Scholar
Yaw, K., Plonsky, L., Larsson, T., Sterling, S., & Kyto, M. (2023). Timeline: Research ethics in applied linguistics. Language Teaching, 56(4), 478–494. Google Scholar logo with link to Google Scholar
Zhang, X. (2020). A bibliometric analysis of second language acquisition between 1997 and 2018. Studies in Second Language Acquisition, 42(1), 199–222. Google Scholar logo with link to Google Scholar
Zhonggen, Y., Ying, Z., Zhichun, Y., & Wentao, C. (2019). Student satisfaction, learning outcomes, and cognitive loads with a mobile learning platform. Computer assisted language learning, 32(4), 323–341. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue