Article published In: English Text Construction
Vol. 13:1 (2020) ► pp.84–107
Assessing writing performance in TOEFL-iBT
A closer look at linguistic and discoursal features
Published online: 24 July 2020
https://doi.org/10.1075/etc.00036.sho
https://doi.org/10.1075/etc.00036.sho
Abstract
This study aims to determine the linguistic and discoursal differences in essays produced by Iranian test-takers of TOEFL-iBT in response to integrated and independent writing tasks. A sample of 40 essays, written by 20 Iranian test-takers of scored integrated and independent writing tasks, was compared and analyzed in terms of the four latent constructs of text easability (fourteen variables), cohesion (nine variables), lexical sophistication (nineteen variables), and syntactic complexity (six variables), using the Coh-Metrix 3.0 program. Results indicate differences in the linguistic and discoursal features of integrated and independent writing tasks. The findings reveal that the scores on writing tasks of EFL test-takers can be anchored empirically through the analysis of some discourse qualities like cohesion. Independent tasks contain more connectives and particles so they can result in better discourse structure organization and the generation of more cohesive devices. Stakeholders of the test should verify test constructs in terms of particular contexts like EFL and communicative views of language proficiency. Consequently, the findings contribute to the ongoing validity argument on TOEFL-iBT writing tasks for designing and interpreting scoring schemes for the writing component of the test.
Article outline
- 1.Introduction
- 2.Background to the study
- 3.The study
- 3.1Data
- 3.2Instrument
- 3.3Data collection procedure and data analysis
- 4.Results
- 5.Discussion and conclusion
References
References (41)
Banerjee, Jayanti, Florencia Franceschina & Anne M. Smith. 2007. Documenting features of written language production typical of different IELTS band score levels (IELTS Research Reports). Canberra, Australia: IELTS Australia Pty and British Council.
Biber, Douglas & Bethany Gray. 2013. Discourse characteristics of writing and speaking task types on the TOEFL-iBT test: A lexico-grammatical analysis. TOEFL-iBT research report. Retrieved from <[URL]> (6 May 2020).
Chapelle, Carol. A., Mary K. Enright & Joan M. Jamieson. 2008. Building a Validity Argument for the Test of English as a Foreign Language TM. New York, NY: Routledge.
Cho, Yeonsuk, Frank Rijmen & Jakub Novak. 2013. Investigating the effects of prompt characteristics on the comparability of TOEFL-iBT integrated writing tasks. Language Testing 30 (4): 513–534.
Connor, Ulla. 1990. Linguistic/rhetorical measures of international persuasive student writing. Research in the Teaching of English 24 (1): 67–87.
Crossley, Scott A., Amanda M. Clevinger & Youjin Kim. 2014. The role of lexical properties and cohesive devices in text integration and their effect on human ratings of speaking proficiency. Language Assessment Quarterly 11(3): 250–270.
Crossley, Scott A., & Danielle S. McNamara. 2012. Predicting second language writing proficiency: The role of cohesion, readability, and lexical difficulty. Journal of Research in Reading 35(2): 115–135.
Crossley, Scott A. & Danielle S. McNamara. 2013. Applications of text analysis tools for spoken response grading. Language Learning & Technology 17(2): 171–192.
Cumming, Alister, Robert Kantor, Kyoko Baba, Keanre Eouanzoui, Usman Erdoosy & Mark James. 2006. Analysis of discourse features and verification of scoring levels for independent and integrated tasks for the new TOEFL. Princeton, NJ: Educational Testing Service.
Cumming, Alister, Robert Kantor, Kyoko Baba, Usman Erdoosy, Keanre Eouanzoui & Mark James. 2005. Differences in written discourse in independent and integrated prototype tasks for next generation TOEFL. Assessing Writing 10(1): 5–43.
Cumming, Alister, Robert Kantor, Donald E. Powers, Terry Santos & Carol Taylor. 2000. TOEFL 2000 writing framework: A working paper. In TOEFL-MS-18. Princeton, NJ: Educational Testing Service.
Deane, Paul. 2013. On the relation between automated essay scoring and modern views of the writing construct. Assessing Writing 18(1): 7–24.
Educational Testing Service. 2018. Reliability and Comparability of TOEFL-iBT Scores. Princeton, NJ: Educational Testing Service.
Engber, Cheryl. A. 1995. The relationship of lexical proficiency to the quality of ESL compositions. Journal of Second Language Writing 4(2): 139–155.
Ferris, Dana R. 1994. Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly 28(2): 414–420.
Frase, Lawrence T., Joseph Faletti, April Ginther & Leslie Grant. 1999. Computer Analysis of the TOEFL Test of Written English [TOEFL Research Report No. 64]. Princeton, NJ: ETS.
Gebril, Atta & Lia Plakans. 2009. Investigating source use, discourse features, and process in integrated writing tests. Spaan Fellow Working Papers in Second/foreign Language Assessment 71: 47–84.
Graesser, Arthur C. & Danielle S. McNamara. 2011. Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science 3 (2): 371–398.
Graesser, Arthur C., Danielle S. McNamara, Max M. Louwerse & Zhiqiang Cai. 2004. Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers 36(2): 193–202.
Graesser, Arthur C., Murray Singer & Tom Trabasso. 1994. Constructing inferences during narrative text comprehension. Psychological Review 101(3): 371–395.
Grant, Leslie & April Ginther. 2000. Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing 9(2): 123–145.
Guo, Liang. 2011. Product and process in TOEFL-iBT independent and integrated writing tasks: An investigation of construct validity. Unpublished doctoral dissertation, Georgia State University.
Guo, Liang, Scott A. Crossley & Danielle S. McNamara. 2013. Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing 18(3): 218–238.
Hamp-Lyons, Liz & Barbara Kroll. 1996. Issues in ESL writing assessment: An overview. College ESL 6(1): 52–72.
Jarvis, Scott. 2002. Short texts, best-fitting curves and new measures of lexical diversity. Language Testing 19(1): 57–84.
Jin, Wenjun. 2001. A quantitative study of cohesion in Chinese graduate students’ writing: Variations across genres and proficiency levels (ERIC Document Reproduction Service No. ED 452 726).
Kintsch, Walter. 1998. Comprehension: A paradigm for cognition. New York, NY: Cambridge University Press.
Knoch, Ute, Susy Macqueen & Sally O’Hagan. 2014. An Investigation of the Effect of Task Type on the Discourse Produced by Students at Various Score Levels in the TOEFL-iBT® Writing Test [Research Report No. RR-14–43]. Princeton, NJ: Educational Testing Service.
Knoch, Ute. 2009. Diagnostic Assessment of Writing: The Development and Validation of a Rating Scale. Frankfurt: Peter Lang.
Kyle, Kristopher & Scott A. Crossley. 2016. The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing 341: 12–24.
Landauer, Thomas K., Danielle S. McNamara, Simon Dennis & Walter Kintsch (eds). 2007. LSA: A Road to Meaning. Mahwah, NJ: Lawrence Erlbaum Associates.
Leki, Ilona & Joan Carson. 1997. Completely different worlds: EAP and the writing experiences of ESL students in university courses. TESOL Quarterly 31(1): 36–69.
McNamara, Danielle S., Scott A. Crossley & Philip M. McCarthy. 2010. Linguistic features of writing quality. Written Communication 27(1): 57–86.
McNamara, Danielle S. & Arthur C. Graesser. 2012. Coh-Metrix: An automated tool for theoretical and applied natural language processing. In Applied Natural Language Processing: Identification, Investigation, and Resolution, P. M. McCarthy, & C. Boonthum-Denecke (eds), 188–205. Hershey, PA: IGI Global.
McNamara, Danielle S. & Joe Magliano. 2009. Toward a comprehensive model of comprehension. Psychology of Learning and Motivation – Advances in Research and Theory 511: 297–384.
McNamara, Danielle S., Max M. Louwerse, Philip M. McCarthy & Arthur C. Graesser. 2010. Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes 47(4): 292–330.
Norris, John M. & Lourdes Ortega. 2009. Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics 30(4): 555–578.
Plakans, Lia, Atta Gebril & Zeynep Bilki. 2019. Shaping a score: The impact of fluency, accuracy, and complexity on integrated skills performances. Journal of Language Testing 36 (2): 161–179.
Riazi, Mehdi A. 2016. Comparing writing performance in TOEFL-iBT and academic assignments: An exploration of textual features. Assessing Writing 281, 15–27.
