Cover not available

Article published In: Pedagogical Linguistics: Online-First Articles

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB

Measuring grammatical understanding

Assessing the usability of the Test for Grammatical Understanding (TGU) through item response theory analyses

ORCiD logo with link | Utrecht University
 | Tilburg University
 | University of Antwerp
Open Access logo
Available under the Creative Commons Attribution (CC BY) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Open Access publication of this article was funded through a Transformative Agreement with Utrecht University.

Published online: 18 March 2025
DOI logo with link
https://doi.org/10.1075/pl.24014.van
Abstract
In the realm of first language (L1) grammar education, research on instruments assessing explicit grammatical understanding is limited. This article discusses an Item Response Theory (IRT) analysis on the Test for Grammatical Understanding (TGU), a multiple-choice tool designed for measuring such grammatical understanding, in this case pertaining to the concept of valency and related concepts (e.g., subject, objects). The results draw from 189 pre-university students (aged 14), who completed different versions of the TGU at multiple time points. Using IRT models for categorical data, we assessed item difficulty, discrimination, and response category functioning. Results highlight the TGU’s usability, version interchangeability, and its potential as a tool for monitoring progress in grammatical understanding. Additionally, differential item functioning (DIF) analyses indicated that most test items performed consistently across versions, though a few showed discrepancies requiring further examination. The discussion addresses item success variations, proposing items that might be omitted under time constraints, and examines the value of a ‘partial understanding’ answer category alongside categories reflecting ‘full’ and ‘no understanding.’ Future study recommendations include expanding TGUs to different linguistic concepts beyond valency and exploring its correlation with open reasoning tasks to further validate its effectiveness in assessing explicit grammatical understanding.
Article outline
  • 1.Introduction
  • 2.The origins of the TGU
  • 3.The construction of the TGU
  • 4.Research projects using the TGU
  • 5.Aim of the current study
  • 6.Method
    • 6.1Participants
    • 6.2Statistical analyses
  • 7.Results
    • 7.1Descriptive statistics
    • 7.2Estimating the best model for the data
    • 7.3Differential item functioning
  • 8.Discussion
    • 8.1Explaning poorly discriminating test items
    • 8.2Recommendations for educational practice and future research
  • References
References (56)
References
Alderson, J., & Hudson, R. (2013). The metalinguistic knowledge of undergraduate students of English language or linguistics. Language Awareness, 22(4), 320–337. Google Scholar logo with link to Google Scholar
Arseneau, R., Foucambert, D., & Lefrançois, P. (2018). Improving the mastery of relative clause in French L1 secondary classes: The effects of an intervention based on verbal interactions on written syntactic structures. L1-Educational Studies in Language and Literature, 18(1), 1–29. Google Scholar logo with link to Google Scholar
Baumberger, C. (2019). Explicating objectual understanding: Taking degrees seriously. Journal for General Philosophy of Science, 501, 367–388. Google Scholar logo with link to Google Scholar
Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how? CBE — Life Sciences Education, 15(4), 1–7. Google Scholar logo with link to Google Scholar
Boivin, M.-C., Fontich, X., Funke, R., García-Folgado, M.-J., & Myhill, D. (2018). Working on grammar at school in L1 education: Empirical research across linguistic regions. L1-Educational Studies in Language and Literature, 18(3), 1–6. Google Scholar logo with link to Google Scholar
Brøseth, H., & Nygård, M. (2023). Norwegian first-year student teachers’ knowledge of L1 grammar. L1-Educational Studies in Language and Literature, 23(1), 1–30. Google Scholar logo with link to Google Scholar
Broekhuis, H., Corver, N. & Vos, R. (2015). Syntax of Dutch: Verbs and Verb Phrases. Volume 1. Amsterdam University Press.Google Scholar logo with link to Google Scholar
Cai, L., & Hansen, M. (2013). Limited-information goodness-of-fit testing of hierarchical item factor models. British Journal of Mathematical and Statistical Psychology, 661, 245–276. Google Scholar logo with link to Google Scholar
Chalmers, R. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. Google Scholar logo with link to Google Scholar
Chen, H., & Myhill, D. (2016). Children talking about writing: Investigating metalinguistic understanding. Linguistics and Education, 351, 100–108. Google Scholar logo with link to Google Scholar
Choi, S. W., Gibbons, L. E., & Crane, P. K. (2016). lordif: Logistic ordinal regression differential item functioning using IRT (Version 0.3–3) [R package]. [URL]
Coppen, P.-A., Van Rijt, J., Wijnands, A., & Dielemans, R. (2019). Vakdidactisch onderzoek naar grammaticaonderwijs in Nederland. Tijdschrift voor Nederlandse Taal- en Letterkunde, 135(2), 85–99. Google Scholar logo with link to Google Scholar
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 161, 297–334. Google Scholar logo with link to Google Scholar
Fenwick, L., Humphrey, S., Quinn, M., & Endicott, M. (2013). Developing deep understanding about language in undergraduate pre-service teacher programs through the application of knowledge. Australian Journal of Teacher Education, 39(1), 1–38. Google Scholar logo with link to Google Scholar
Fontich, X. (2014). Grammar and language reflection at school: Checking out the whats and the hows of grammar instruction. In T. Ribas, X. Fontich, & O. Guasch (Eds.), Grammar at school: Research on metalinguistic activity in language education (pp. 255–284). Peter Lang.Google Scholar logo with link to Google Scholar
Fontich, X., & Camps, A. (2014). Towards a rationale for research into grammar teaching at schools. Research Papers in Education, 29(5), 598–625. Google Scholar logo with link to Google Scholar
Fontich, X., Van Rijt, J., & Gauvin, I. (2020). Research on L1 grammar in schooling: Mediation at the heart of learning grammar. L1-Educational Studies in Language and Literature, 20(3), 1–13. Google Scholar logo with link to Google Scholar
Giovanelli, M. (2015). Becoming an English language teacher: Linguistic knowledge, anxieties and the shifting sense of identity. Language and Education, 29(5), 416–429. Google Scholar logo with link to Google Scholar
Graham, S., Kim, Y.-S., Cao, Y., Lee, J., Tate, T., Collins, P., Cho, M., Moon, Y., Chung, H. Q., & Olson, C. B. (2023). A meta-analysis of writing treatments for students in grades 6–12. Journal of Educational Psychology, 115(7), 1004–1027. Google Scholar logo with link to Google Scholar
Hmelo-Silver, C. E., & Pfeffer, M. (2004). Comparing expert and novice understanding of a complex system from the perspective of structures, behaviors, and functions. Cognitive Science, 281, 127–138. Google Scholar logo with link to Google Scholar
Land, R., Meyer, J. H. F., & Smith, J. (2008). Threshold concepts within the disciplines. BRILL. Google Scholar logo with link to Google Scholar
Luif, J. (1994). In verband met de zin: Inleiding in de Nederlandse Spraakkunst (3rd ed.). Amsterdam University Press.Google Scholar logo with link to Google Scholar
Instituut voor de Nederlandse Taal (2021). Algemene Nederlandse Spraakkunst. Third, revised version.Google Scholar logo with link to Google Scholar
Jones, P., & Chen, H. (2012). Teachers’ knowledge about language: Issues of pedagogy and expertise. Australian Journal of Language and Literacy, 35(2), 147–168. Google Scholar logo with link to Google Scholar
Leenders, G., de Graaff, H., & van Koppen, J. M. (2023). Hoe meet je bewuste taalvaardigheid? Grammaticaal redeneren in de vakken Nederlands, Engels en Duits. Pedagogische Studiën, 98(1). Google Scholar logo with link to Google Scholar
Locke, T. (Ed.). (2010). Beyond the grammar wars: A resource for teachers and students on developing language knowledge in the English/literacy classroom. Routledge. Google Scholar logo with link to Google Scholar
Maydeu-Olivares, A., & Joe, H. (2014). Assessing Approximate Fit in Categorical Data Analysis. Multivariate Behavioral Research, 49(4), 305–328. Google Scholar logo with link to Google Scholar
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 471, 149–174. Google Scholar logo with link to Google Scholar
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 161, 159–176. Google Scholar logo with link to Google Scholar
Myhill, D. (2000). Misconceptions and difficulties in the acquisition of metalinguistic knowledge. Language and Education, 14(3), 151–163. Google Scholar logo with link to Google Scholar
(2003). Principled understanding? Teaching the active and passive voice. Language and Education, 17(5), 355–370. Google Scholar logo with link to Google Scholar
(2018). Grammar as a meaning-making resource for language development. L1-Educational Studies in Language and Literature, 18(3), 1–21. Google Scholar logo with link to Google Scholar
(2021). Grammar re-imagined: Foregrounding understanding of language choice in writing. English in Education, 55(3), 265–278. Google Scholar logo with link to Google Scholar
Myhill, D., Jones, S., & Lines, H. (2018). Supporting less proficient writers through linguistically aware teaching. Language and Education, 32(4), 333–349. Google Scholar logo with link to Google Scholar
Myhill, D., Jones, S., & Watson, A. (2013). Grammar matters: How teachers’ grammatical knowledge impacts on the teaching of writing. Teaching and Teacher Education, 361, 77–91. Google Scholar logo with link to Google Scholar
Perini, M. (2015). Describing verb valency: Practical and theoretical issues. Springer. Google Scholar logo with link to Google Scholar
Rasch, G. (1960). Probabilistic models for some intelligence and achievement tests. Danish Institute for Educational Research.Google Scholar logo with link to Google Scholar
Rättyä, K., Awramiuk, E., & Fontich, X. (2019). What is grammar in L1 education today? L1-Educational Studies in Language and Literature, 19(2), 1–8. Google Scholar logo with link to Google Scholar
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, 171. Google Scholar logo with link to Google Scholar
Sangster, P., Anderson, C., & O’Hara, P. (2013). Perceived and actual levels of knowledge about language amongst primary and secondary student teachers: Do they know what they think they know? Language Awareness, 22(4), 293–319. Google Scholar logo with link to Google Scholar
Taber, K. S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 481, 1273–1296. Google Scholar logo with link to Google Scholar
Trotzke, A. (2023). Pedagogical linguistics: Connecting formal linguistics to language teaching. Language, 99(3), 153–175. Google Scholar logo with link to Google Scholar
Van Gelderen, A. (2010). Does explicit teaching of grammar help students to become better writers? Insights from empirical research. In T. Locke (Ed.), Beyond the grammar wars: A resource for teachers and students on developing language knowledge in the English/literacy classroom (pp. 109–128). Routledge.Google Scholar logo with link to Google Scholar
Stadler, M., Sailer, M., & Fischer, F. (2021). Knowledge as a formative construct: A good alpha is not always better. New Ideas in Psychology, 601. Google Scholar logo with link to Google Scholar
Van Rijt, J. (2020). Understanding grammar: the impact of linguistic metaconcepts on L1 grammar education. Doctoral dissertation. Radboud University Nijmegen.
(2024). Learning how to think like a linguist: Linguistic reasoning as a focal point in L1 grammar education. Language and Linguistics Compass, 18(3), 1–17. Article e12513. Google Scholar logo with link to Google Scholar
Van Rijt, J., & Coppen, P.-A. J. M. (2021). The conceptual importance of grammar: Knowledge-related rationales for grammar teaching. Pedagogical Linguistics, 2(2), 175–199. [URL].
Van Rijt, J., Banga, A., & Goudbeek, M. (2024). Getting a load of linguistic reasoning: How L1 student teachers process rules of thumb and linguistic manipulations in discussions about grammar. Applied Linguistics, 45(1), 163–188. Google Scholar logo with link to Google Scholar
Van Rijt, J. H. M., de Swart, P., Wijnands, A., & Coppen, P.-A. (2019). When students tackle grammatical problems: Exploring linguistic reasoning with linguistic metaconcepts in L1 grammar education. Linguistics and Education, 521, 78–88. Google Scholar logo with link to Google Scholar
Van Rijt, J., Hulshof, H., & Coppen, P.-A. (2021). ‘X is the odd one out, because the other two are more about the farmland’: Dutch L1 student teachers’ struggles to reason about grammar in odd one out tasks. Studies in Educational Evaluation, 701, Article 101007. Google Scholar logo with link to Google Scholar
Van Rijt, J., Myhill, D., De Maeyer, S., & Coppen, P.-A. (2022). Linguistic metaconcepts can improve grammatical understanding in L1 education evidence from a Dutch quasi-experimental study. PLoS One, 17(2), Article e0263123. Google Scholar logo with link to Google Scholar
Van Rijt, J., Wijnands, A., & Coppen, P.-A. (2020). How secondary school students may benefit from linguistic metaconcepts to reason about L1 grammatical problems. Language and Education, 34(3), 231–248. Google Scholar logo with link to Google Scholar
Watson, A., & Newman, R. (2017). Talking grammatically: L1 adolescent metalinguistic reflection on writing. Language Awareness, 26(4), 381–398. Google Scholar logo with link to Google Scholar
Wijnands, A., van Rijt, J., Stoel, G., & Coppen, P.-A. (2022). Balancing between uncertainty and control: Teaching reflective thinking about language in the classroom. Linguistics and Education, 711, 1–19. Google Scholar logo with link to Google Scholar
Xu, J., Paek, I., & Xia, Y. (2017). Investigating the Behaviors of M2 and RMSEA2 in Fitting a Unidimensional Model to Multidimensional Data. Applied Psychological Measurement, 41(8), 632–644. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue