Cover not available

Article published In: Chinese Language and Discourse
Vol. 11:2 (2020) ► pp.169–197

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Incoherence in L2 writing

A comparison of expert insider and non-expert outsider ratings

 | Arizona State University
Published online: 24 November 2020
DOI logo with link
https://doi.org/10.1075/cld.20018.lia
Abstract
Current knowledge of the functional dimensions (e.g., coherence) of L2 written performance is mainly based on expert readers’ views. Non-expert native readers’ perspectives of L2 written productions are not often examined, which prevents a comprehensive and objective understanding of how an L2 writer’s performance may be perceived and evaluated by the target discourse community. Studies to date also lack clear findings of both the types of incoherence phenomena that may exist in L2 texts and the factors that may contribute to such incoherencies. The current study investigates how expert and non-expert native readers evaluate incoherence in L2 Chinese argumentative essays, as well as how their evaluations may differ quantitatively or qualitatively. The findings reveal that although expert readers marked incoherence in L2 writing significantly more frequently than non-expert readers did, expert and non-expert readers displayed similar patterns and tendencies in their judgments pertaining to the frequency with which incoherence instances appeared in the essays and in the severity of the incoherence.
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1Conceptualizations of L2 written coherence
    • 2.2Expert and non-expert raters’ judgments of L2 written coherence
    • 2.3Development of L2 written coherence
  • 3.The present study
    • 3.1Dataset
    • 3.2Raters
    • 3.3Rating
    • 3.4Analysis
  • 4.Results
    • 4.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
    • 4.2Reasons for incoherence by the expert and non-expert raters
    • 4.3Inter-rater relationships
  • 5.Discussion
    • 5.1Judgments of frequency and severity of incoherence by the expert and non-expert raters
    • 5.2Reasons for incoherence by the expert and non-expert raters
  • 6.Conclusion and implications
  • Reference
References (45)
Reference
Bardovi-Harlig, K. 1990. Pragmatic word order in English composition. In U. Connor & A. Johns (Eds.), Coherence in writing: Research and pedagogical perspectives (pp. 43–65). Alexandria, VA: TESOL.Google Scholar logo with link to Google Scholar
Carrell, P. 1982. Cohesion is not coherence. TESOL Quarterly, 161, 479–488. Google Scholar logo with link to Google Scholar
Chen, Y., & Baker, P. 2016. Investigating criterial discourse features across second language development: Lexical bundles in rated learner essays, CEFR B1, B2 and C1. Applied Linguistics, 371, 849–880.Google Scholar logo with link to Google Scholar
Chiang, S. 2003. The importance of cohesive conditions to perceptions of writing quality at the early stages of foreign language learning. System, 311, 471–484. Google Scholar logo with link to Google Scholar
Connor, U. 1990. Linguistic/rhetorical measures for international persuasive student writing. Research in the Teaching of English, 241, 67–87.Google Scholar logo with link to Google Scholar
Crossley, S., Kyle, K., & McNamara, D. 2016a. The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 481, 1227–1237. Google Scholar logo with link to Google Scholar
2016b. The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 321, 1–16. Google Scholar logo with link to Google Scholar
Crossley, S., & McNamara, D. 2011. Shared features of L2 writing: Intergroup homogeneity and text classification. Journal of Second Language Writing, 201, 271–285. Google Scholar logo with link to Google Scholar
2012. Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35, 2, 115–135. Google Scholar logo with link to Google Scholar
Crossley, S., Yang, H. S., & McNamara, D. 2014. What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 26(1), 92–113.Google Scholar logo with link to Google Scholar
Cumming, A. 1990. Expertise in evaluating second language compositions. Language Testing, 71, 31–51. Google Scholar logo with link to Google Scholar
Eckes, T. 2008. Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155–185. Google Scholar logo with link to Google Scholar
Ferris, D. 1994. Lexical and syntactic features of ESL writing by students at different levels of L2 proficiency. TESOL Quarterly, 281, 414–420. Google Scholar logo with link to Google Scholar
Guo, L., Crossley, S., & McNamara, D. 2013. Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 181, 218–238. Google Scholar logo with link to Google Scholar
Halliday, M., & Hasan, R. 1976. Cohesion in English. London, UK: Longman.Google Scholar logo with link to Google Scholar
Harman, R. 2013. Literary intertextuality in genre-based pedagogies: Building lexical cohesion in fifth-grade L2 writing. Journal of Second Language Writing, 221, 125–140. Google Scholar logo with link to Google Scholar
Hyland, K. 2005. Metadiscourse: Exploring interaction in writing. London, UK: Continuum.Google Scholar logo with link to Google Scholar
Jiang, W. 2013. Measurements of development in L2 written production: The case of L2 Chinese. Applied Linguistics, 341, 1–24. Google Scholar logo with link to Google Scholar
Kang, Y. J. 2005. Written narratives as an index of L2 competence in Korean EFL learners. Journal of Second Language Writing, 141, 259–279. Google Scholar logo with link to Google Scholar
Kennedy, C., & Thorp, D. 2007. A corpus-based investigation of linguistic responses to an IELTS academic writing task. In L. Taylor & P. Falvey (Eds), IELTS collected papers: Research in speaking and writing assessment (pp. 316–377). Cambridge, UK: Cambridge University Press.Google Scholar logo with link to Google Scholar
Knoch, U. 2007. ‘Little coherence, considerable strain for reader’: A comparison between two rating scales for the assessment of coherence. Assessing Writing, 12(2), 108–128. Google Scholar logo with link to Google Scholar
Kormos, J. 2011. Task complexity and linguistic and discourse features of narrative writing performance. Journal of Second Language Writing, 201, 148–161. Google Scholar logo with link to Google Scholar
Kuiken, F., & Vedder, I. 2014. Rating written performance: What do raters do and why? Language Testing, 311, 329–348. Google Scholar logo with link to Google Scholar
2017. Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 341, 321–336. Google Scholar logo with link to Google Scholar
Kuiken, F., Vedder, I., & Gilabert, R. 2010. Communicative adequacy and linguistic complexity in L2 writing. In I. Bartning, M. Martin & I. Vedder (Eds.), Communicative proficiency and linguistic development: Intersections between SLA and language testing research (pp. 81–99). Eurosla: European Second Language Association.Google Scholar logo with link to Google Scholar
Li, S. 2014. The gap in the use of lexical cohesive devices in writing between native Chinese speakers and second language users. Journal of the Chinese Language Teachers Association, 49(3), 25–47.Google Scholar logo with link to Google Scholar
Li, T., & Wharton, S. 2012. Metadiscourse repertoire of L1 Mandarin undergraduates writing in English: A cross-contextual, cross-disciplinary study. Journal of English for Academic Purposes, 111, 345–356. Google Scholar logo with link to Google Scholar
Liu, M., & Braine, G. 2005. Cohesive features in argumentative writing produced by Chinese undergraduates. System, 331, 623–636. Google Scholar logo with link to Google Scholar
McCulley, G. 1985. Writing quality, coherence, and cohesion. Research in the Teaching of English, 191, 269–282.Google Scholar logo with link to Google Scholar
McCutchen, D. 1996. A capacity theory of writing: Working memory in composition. Educational Psychology Review, 81, 299–325. Google Scholar logo with link to Google Scholar
McNamara, D. 2001. Reading both high and low coherence texts: Effects of text sequence and prior knowledge. Canadian Journal of Experimental Psychology, 551, 51–62. Google Scholar logo with link to Google Scholar
McNamara, D., Crossley, S., & McCarthy, P. 2010. Linguistic features of writing quality. Written Communication, 271, 57–86. Google Scholar logo with link to Google Scholar
McNamara, D., Crossley, S., & Roscoe, R. 2013. Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 451, 499–515. Google Scholar logo with link to Google Scholar
McNamara, D., & Kintsch, W. 1996. Learning from texts: Effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288. Google Scholar logo with link to Google Scholar
McNamara, D., Kintsch, E., Songer, N., & Kintsch, W. 1996. Are good texts always better? Interactions of text coherence, background knowledge, and levels of understanding in learning from text. Cognition and Instruction, 141, 1–43. Google Scholar logo with link to Google Scholar
Neuner, J. 1987. Cohesive ties and chains in good and poor freshman essays. Research in the Teaching of English, 211, 92–105.Google Scholar logo with link to Google Scholar
O’Reilly, T., & McNamara, D. 2007. Reversing the reverse cohesion effect: Good texts can be better for strategic, high-knowledge readers. Discourse Processes, 431, 121–152. Google Scholar logo with link to Google Scholar
Reid, J. 1992. A computer text analysis of four cohesion devices in English discourse by native and nonnative writers. Journal of Second Language Writing, 11, 79–107. Google Scholar logo with link to Google Scholar
Schoonen, R. 2005. Generalizability of writing scores: an application of structural equation modeling. Language Testing, 22(1), 1–30. Google Scholar logo with link to Google Scholar
Schoonen, R., Vergeer, M., & Eiting, M. 1997. The assessment of writing ability: Expert readers versus lay readers. Language Testing, 14(2), 157–184. Google Scholar logo with link to Google Scholar
Tannen, D. 1984. Spoken and written narrative in English and Greek. In D. Tannen (Ed.). Coherence in spoken and written discourse (pp. 21–41). Norwood, NJ: Ablex.Google Scholar logo with link to Google Scholar
Weigle, S. C. 2007. Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194–209. Google Scholar logo with link to Google Scholar
Witte, S., & Faigley, L. 1981. Coherence, cohesion, and writing ability. College Composition and Communication, 321, 189–204. Google Scholar logo with link to Google Scholar
Yang, C. 2013. Textual conjunctives and topic-fronting devices in CFL learners’ written summaries. Journal of the Chinese Language Teachers Association, 48(1), 71–89.Google Scholar logo with link to Google Scholar
Yang, W., & Sun, Y. 2012. The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and Education, 231, 31–48. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue