Cover not available

Article published In: Words in the World
Edited by Laura Teddiman, Lori Buchanan and Hamad Al-Azary
[The Mental Lexicon 19:1] 2024
► pp. 90–99

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

NLP and education

Using semantic similarity to evaluate filled gaps in a large-scale Cloze test in the classroom

ORCiD logo with link | Federal University of Sergipe
ORCiD logo with link | Federal University of Sergipe
ORCiD logo with link | Federal University of Sergipe
ORCiD logo with link | Federal University of Sergipe
Published online: 10 January 2025
DOI logo with link
https://doi.org/10.1075/ml.24027.deg
Abstract
This study examines the applicability of the Cloze test, a widely used tool for assessing text comprehension proficiency, while highlighting its challenges in large-scale implementation. To address these limitations, an automated correction approach was proposed, utilizing Natural Language Processing (NLP) techniques, particularly word embeddings (WE) models, to assess semantic similarity between expected and provided answers. Using data from Cloze tests administered to students in Brazil, WE models for Brazilian Portuguese (PT-BR) were employed to measure the semantic similarity of the responses. The results were validated through an experimental setup involving twelve judges who classified the students’ answers. A comparative analysis between the WE models’ scores and the judges’ evaluations revealed that GloVe was the most effective model, demonstrating the highest correlation with the judges’ assessments. This study underscores the utility of WE models in evaluating semantic similarity and their potential to enhance large-scale Cloze test assessments. Furthermore, it contributes to educational assessment methodologies by offering a more efficient approach to evaluating reading proficiency.
Article outline
  • Introduction
  • 2.The Cloze procedure
  • 3.Semantic similarity
  • 4.Method
    • 4.1Participants
    • 4.2Cloze test procedure
    • 4.3Assessment of similarity by humans
    • 4.4Assessment of similarity by word embeddings models
    • 4.5Validation
  • 5.Results
  • 6.Conclusion
  • Acknowledgements
  • Note
  • References
References (19)
References
Bickley, A. C., Ellington, B. J., & Bickley, R. T. (1970). The cloze procedure: A conspectus. Journal of Reading Behavior, 2(3), 232–249. Google Scholar logo with link to Google Scholar
Brown, J. D. (2002). Do cloze tests work? Or is it just an illusion?. Second Language Studies, 21 (1), 79–125.Google Scholar logo with link to Google Scholar
(1980). Relative merits of four methods for scoring cloze tests. The Modern Language Journal, 64(3), 311–317. Google Scholar logo with link to Google Scholar
Cardoso, P. B., Menezes, K. V., Freitas, F. O., & Freitag, R. M. K. (2024). Eficiência na leitura: medidas de precisão e velocidade entre alunos do Colégio de Aplicação da Universidade Federal de Sergipe. Revista Científica Sigma, 5(5), 120–143.Google Scholar logo with link to Google Scholar
Chandrasekaran, D., & Mago, V. (2021). Evolution of semantic similarity — a survey. ACM Computing Surveys (CSUR), 54(2), 1–37. Google Scholar logo with link to Google Scholar
Cunha, N. D. B., & Santos, A. A. A. D. (2010). Estudos de validade entre instrumentos que avaliam habilidades linguísticas. Estudos de Psicologia (Campinas), 271, 305–314. Google Scholar logo with link to Google Scholar
Darnell, D. K. (1968). The Development of an English Language Proficiency Test of Foreign Students, Using a Clozentropy Procedure. Final Report.Google Scholar logo with link to Google Scholar
Gorman, J., & Curran, J. R. (2006, July). Scaling distributional similarity to large corpora. In Proceedings of the 21 International Conference on Computational Linguistics and 44 Annual Meeting of the Association for Computational Linguistics (pp. 361–368). Google Scholar logo with link to Google Scholar
Hartmann, N., Fonseca, E., Shulby, C., Treviso, M., Rodrigues, J., & Aluisio, S. (2017). Portuguese word embeddings: Evaluating on word analogies and natural language tasks. arXiv preprint arXiv:1708.06025.Google Scholar logo with link to Google Scholar
Lange, K., Kühn, S., & Filevich, E. (2015). “Just another tool for online studies” (JATOS): An easy solution for setup and management of web servers supporting online studies. PloS one, 10(6), e0130834. Google Scholar logo with link to Google Scholar
Levy, O., & Goldberg, Y. (2014). Neural word embedding as implicit matrix factorization. Advances in neural information processing systems, 271.Google Scholar logo with link to Google Scholar
Ling, W., Dyer, C., Black, A. W., & Trancoso, I. (2015). Two/too simple adaptations of word2vec for syntax problems. In Proceedings of the 2015 conference of the North American chapter of Association for Computational Linguistics: human language technologies (pp. 1299–1304). Google Scholar logo with link to Google Scholar
Lowry, D. T., & Marr, T. J. (1975). Clozentropy as a measure of international communication comprehension. Public Opinion Quarterly, 39(3), 301–312. Google Scholar logo with link to Google Scholar
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.Google Scholar logo with link to Google Scholar
Mohammad, S. M., & Hirst, G. (2012). Distributional measures of semantic distance: A survey. arXiv preprint arXiv:1203.1858.Google Scholar logo with link to Google Scholar
Oller Jr, J. W., & Conrad, C. A. (1971). The Cloze technique and ESL proficiency. Language Learning, 21(2), 183–194. Google Scholar logo with link to Google Scholar
Pennington, J., Socher, R., & Manning, C. D. (2014, October). Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543). Google Scholar logo with link to Google Scholar
Taylor, W. L. (1953). “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 30(4), 415–433. Google Scholar logo with link to Google Scholar
Wobbrock, J. O., Findlater, L., Gergle, D., & Higgins, J. J. (2011, May). The aligned rank transforms for nonparametric factorial analyses using ANOVAanova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 143–146). Google Scholar logo with link to Google Scholar
Cited by (1)

Cited by one other publication

Gois, Túlio Sousa de & Raquel Meister Ko. Freitag
2025. Anais do XVI Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL 2025), ► pp. 161 ff. DOI logo

This list is based on CrossRef data as of 20 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue