Article published In: ITL - International Journal of Applied Linguistics
Vol. 166:2 (2015) ► pp.278–306
Assessing vocabulary size through multiple-choice formats
Issues with guessing and sampling rates
Published online: 21 January 2016
https://doi.org/10.1075/itl.166.2.04gyl
https://doi.org/10.1075/itl.166.2.04gyl
In most tests of vocabulary size, knowledge is assessed through multiple-choice formats. Despite advantages such as ease of scoring, multiple-choice tests (MCT) are accompanied with problems. One of the more central issues has to do with guessing and the presence of other construct-irrelevant strategies that can lead to overestimation of scores. A further challenge when designing vocabulary size tests is that of sampling rate. How many words constitute a representative sample of the underlying population of words that the test is intended to measure? This paper addresses these two issues through a case study based on data from a recent and increasingly used MCT of vocabulary size: the Vocabulary Size Test. Using a criterion-related validity approach, our results show that for multiple-choice items sampled from this test, there is a discrepancy between the test scores and the scores obtained from the criterion measure, and that a higher sampling rate would be needed in order to better represent knowledge of the underlying population of words. We offer two main interpretations of these results, and discuss their implications for the construction and use of vocabulary size tests.
References (44)
Alderson, J.C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. London: Continuum.
Bachman, L.F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
Beglar, D. (2010). A Rasch-based validation of the Vocabulary Size Test. Language Testing, 27(1), 101–118.
Elgort, I. (2011). Deliberate learning and vocabulary acquisition in a second language. Language Learning, 61(2), 367–413.
. (2013). Effects of L1 definitions and cognate status of test items on the Vocabulary Size Test. Language Testing, 30(2), 253–272.
Eyckmans, J. (2004). Measuring receptive vocabulary size: Reliability and validity of the yes/no vocabulary test for French-speaking learners of Dutch. Utrecht: LOT.
Gyllstad, H. (2012). Validating the Vocabulary Size Test. A classical test theory approach. Poster presented at The Ninth Annual Conference of EALTA, Innsbruck, Austria, 31 May – 3 June. Retrieved from: <[URL]>
Horst, M., Cobb, T., & Meara, P. (1998). Beyond A Clockwork Orange: Acquiring second language vocabulary through reading. Reading in a Foreign Language, 111, 207–223.
Huibregtse, I., Admiraal, W., & Meara, P. (2002). Scores on a yes-no vocabulary test: Correction for guessing and response style. Language Testing, 19(3), 227–245.
Kamimoto, T. (2008). Nation’s vocabulary levels test and its successors: A re-appraisal. Unpublished PhD Thesis. University of Wales, Swansea.
Kane, M.T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73.
Laufer, B., & Goldstein, Z. (2004). Testing vocabulary knowledge: Size, strength, and computer adaptiveness. Language Learning, 54(3), 399–436.
. (1996). The dimensions of lexical competence. In G. Brown, K. Malmkjaer, & J. Williams (Eds.), Performance and competence in second language acquisition (pp. 35–53). Cambridge: Cambridge University Press.
Meara, P., & Buxton, B. (1987). An alternative to multiple choice vocabulary tests. Language Testing, 4(2), 142–154.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Nation, I.S.P. (2006). How large a vocabulary is needed for reading and listening? Canadian Modern Language Review/La Revue Canadienne Des Langues Vivantes, 63(1), 59–82.
Nation, P. (1993). Using dictionaries to estimate vocabulary size: essential, but rarely followed, procedures. Language Testing, 10(1), 27–40.
Nation, I.S.P. (2012). Vocabulary size test instructions and description. Retrieved from <[URL]>. Last revised on October 23, 2012.
Paul, P.V., Stallman, A.C., & O’Rourke, J.P. (1990). Using three test formats to assess good and poor reader’s word knowledge. Technical Report No. 509 of the Center for the Study of Reading. University of Illinois. Retrieved from <[URL]>
Pellicer-Sánchez, A., & Schmitt, N. (2012). Scoring Yes–No vocabulary tests: Reaction time vs. nonword approaches. Language Testing, 29(4), 489–509.
Read, J. (1998). Validating a test to measure depth of vocabulary knowledge. In A.J. Kunnan (Ed.) Validation in language assessment (pp.41–61). Mahwah, NJ: Lawrence Erlbaum Associates.
Rupp, A.A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23(4), 441–474.
Saragi, T., Nation, I.S.P., & Meister, F. (1978). Vocabulary learning and reading. System, 61, 72–78.
Schmitt, N. (2010). Researching vocabulary: A vocabulary research manual. Houndmills: Palgrave Macmillan.
Schmitt, N., & Meara, P. (1997). Researching vocabulary through a word knowledge framework: Word associations and verbal suffixes. Studies in Second Language Acquisition, 191, 17–36.
Schmitt, N., Ng, J.W.C., & Garras, J. (2011). The word associates format: Validation evidence. Language Testing, 28(1), 105–126.
Schmitt, N., & Schmitt, D. (2014). A reassessment of frequency and vocabulary size in L2 vocabulary teaching. Language Teaching, 47(4), 484–503.
Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55–88.
Stenius Staehr, L. (2009). Vocabulary knowledge and advanced listening comprehension in English as a foreign language. Studies in Second Language Acquisition, 311, 577–607.
Stewart, J. (2012). A multiple-choice test of active vocabulary knowledge. Vocabulary Learning and Instruction, 1(1), 53–59.
. (2014). Do multiple-choice options inflate estimates of vocabulary size on the VST? Language Assessment Quarterly, 11(3), 271–282.
Stewart, J., & White, D.A. (2011). Estimating guessing effects on the vocabulary levels test for differing degrees of word knowledge. TESOL Quarterly, 45(2), 370–380.
Uden, J., Schmitt, D., & Schmitt, N. (2014). Jumping from the highest graded readers to ungraded novels: Four case studies. Reading in a Foreign Language, 26(1), 1–28.
Waring, R., & Takaki, M. (2003). At what rate do learners learn and retain new vocabulary from reading a graded reader. Reading in a Foreign Language, 15(2), 130–163.
Cited by (94)
Cited by 94 other publications
Boone, Griet & Vanessa De Wilde
2026. Factors affecting foreign language learners’ receptive and productive word knowledge in English, French, German, and
Spanish. ITL - International Journal of Applied Linguistics
Chen, Shuping & Longhai Yu
de la Viña, Inés, Gloria Chamorro & Christina S. Kim
De Wilde, Vanessa
Li, Chen‐Hong
Milliner, Brett, Joshua Matthews & Stuart McLean
Robles-García, Pablo, Ji-young Shin, Jeffrey Stewart & Stuart McLean
Alamer, Abdullah, Ahmed Al Khateeb & Abdulrahman Alshabeb
Gyllstad, Henrik, Tanja Kupisch & Anika Lloyd-Smith
2025. Development and initial validation of a yes/no vocabulary test for North Sámi. ITL - International Journal of Applied Linguistics 176:1 ► pp. 1 ff.
Gyllstad, Henrik, Pia Sundqvist, Elke Peters, Ulrikke Rindal, Gustaf B. Skar & Nasrin Ulfat
Harding, Luke, Tineke Brunfaut, Ari Huhta & Benjamin Kremmel
Hong, Wilson Cheong Hin & Joyce Zhiqing Li
Klemen, Matej
Nguyen, Chi‐Duc
Oshchepkova, E.S., N.A. Kartushina, M.S. Aslanova, M.E. Dmitrieva & Z.V. Airapetyan
Stewart, Jeffrey, Laurence Anthony, Aaron Olaf Batty, Keita Nakamura, Christopher Nicklin, Stuart McLean & Kanako Tomaru
Uchihara, Takumi, Kazuya Saito, Satsuki Kurokawa, Kotaro Takizawa & Yui Suzukida
Zhang, Haiwei & Dian Prasetyo Adi
Chen, Po-Heng, Rachael C. Hulme, Lena M. Blott & Jennifer M. Rodd
Hamada, Akira, Yuko Hoshino, Masumi Kojima, Kazumi Aizawa, Tatsuo Iso & Yuichiro Kobayashi
Milliner, Brett, Kriss Lange, Joshua Matthews & Riko Umeki
Qi, Shiwei, Mark Feng Teng & Ailan Fu
Robles-García, Pablo, Stuart McLean, Jeffrey Stewart, Ji-young Shin & Claudia Helena Sánchez-Gutiérrez
Sevigny, Paul, Lindsay Mack, Lance Stilp & Maiko Berger
Shafi, Sehrish & Sam Hellmuth
Stewart, Jeffrey, Henrik Gyllstad, Christopher Nicklin & Stuart McLean
Stoeckel, Tim, Hung Tan Ha, Duyen Thi Bich Nguyen & Christopher Nicklin
Stoeckel, Tim, Stuart McLean, Young Ae Kim, Yukie Shinhara & Allie Spencer Patterson
Vicary, Anne & Jeanine Treffers-Daller
Zhang, Haiwei, Peng Sun, Yaowaluk Bianglae & Winda Widiawati
Alzahrani, Alaa
Bardel, Camilla, Henrik Gyllstad & Jörgen Tholin
Dang, Thi Ngoc Yen, Duy Van Vu & Thi My Hang Nguyen
Makovskaya, Liliya & Ijobat Juraeva
Robles-García, Pablo, Glen H. Wallace & Claudia Helena Sánchez-Gutiérrez
Stoeckel, Tim, Tomoko Ishii, Young Ae Kim, Hung Tan Ha, Nam Thi Phuong Ho & Stuart McLean
Sun, Ting & Hyunjeong Nam
Tam, Ho I (Anna) & Barry Lee Reynolds
2023. The relationship between extramural English engagement and the vocabulary size of L1 Cantonese speakers in Macau. ITL - International Journal of Applied Linguistics 174:1 ► pp. 49 ff.
Warnby, Marcus, Hans Malmström & Kajsa Yang Hansen
Zhang, Haiwei, Peijian Paul Sun, Xueyan Zhang, Silin Fan & Yunji Zhang
B. Silva, Breno
B. Silva, Breno
Dabbagh, Ali & Mostafa Janebi Enayat
Nall, Matthew, Ari Huhta & Behzad Mansouri
Urdaniz, Raquel Perez & Sophia Skoufaki
Warnby, Marcus
2022. Receptive academic vocabulary knowledge and extramural English involvement – is there a correlation?. ITL - International Journal of Applied Linguistics 173:1 ► pp. 120 ff.
Warnby, Marcus
Yunjiu, Luo, Wei Wei & Ying Zheng
Zeng, Yuntao, Qiuxia Lu, Matthew P. Wallace, Yawei Guo, Chun-Wai Fan & Xiaofei Chen
Zhang, Songshan & Xian Zhang
Busby, Nicole Louise
Busby, Nicole Louise
Gyllstad, Henrik, Stuart McLean & Jeffrey Stewart
Ha, Hung Tan
SALMELA, ROSA, MINNA LEHTONEN, STEFANO GARUSI & RAYMOND BERTRAM
Schmitt, Norbert, Karen Dunn, Barry O’Sullivan, Laurence Anthony & Benjamin Kremmel
Stewart, Jeffrey, Tim Stoeckel, Stuart McLean, Paul Nation & Geoffrey G. Pinchbeck
Stoeckel, Tim, Stuart McLean & Paul Nation
Dong, Yang, Yi Tang, Sammy Xiao-Ying Wu, Wei-Yang Dong & Zhen Li
Duff, Dawna & Melissa Brydon
Koizumi, Rie & Yo In’nami
McLean, Stuart, Jeffrey Stewart & Aaron Olaf Batty
Reynolds, Barry Lee, Wei-Hua Wu & Ying-Chun Shih
Robles-García, Pablo
Robles‐García, Pablo
Schmitt, Norbert, Paul Nation & Benjamin Kremmel
Zhang, Pengchong & Suzanne Graham
Zhang, Xian, Jianda Liu & Haiyang Ai
Aviad-Levitzky, Tami, Batia Laufer & Zahava Goldstein
Masrai, Ahmed
Masrai, Ahmed
Peters, Elke, Tom Velghe & Tinne Van Rompaey
Puimège, Eva & Elke Peters
Silva, Breno B. & Agnieszka Otwinowska
Stoeckel, Tim, Jeffrey Stewart, Stuart McLean, Tomoko Ishii, Brandon Kramer & Yuko Matsumoto
Janebi Enayat, Mostafa & Esmat Babaii
Masrai, Ahmed & James Milton
Masrai, Ahmed & James Milton
McLean, Stuart
Peters, Elke & Stuart Webb
Petrescu, Maria Claudia & Rena Helms-Park
Reynolds, Barry Lee, Ying-Chun Shih & Wei-Hua Wu
Roche, Thomas & Michael Harrington
Stoeckel, Tim & Tomomi Sukigara
LAUFER, BATIA & TAMI AVIAD–LEVITZKY
Mainz, Nina, Zeshu Shao, Marc Brysbaert & Antje S. Meyer
Stewart, Jeffrey, Stuart McLean & Brandon Kramer
Vilkaitė, Laura
2017. Incidental acquisition of collocations in L2. ITL - International Journal of Applied Linguistics 168:2 ► pp. 248 ff.
İnözü, Julide & Sehnaz Sahinkarakas
Kremmel, Benjamin
Stoeckel, Tim, Phil Bennett & Stuart Mclean
Z Szabo, Csaba
[no author supplied]
This list is based on CrossRef data as of 30 march 2026. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
