Cover not available

Article published In: Journal of Second Language Studies
Vol. 8:1 (2025) ► pp.146–167

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Assessing Chinese EFL learners’ speaking proficiency levels

The distinguishing and predictive roles of cohesive devices

 | Nanjing University, China
 | Nanjing University, China
Published online: 11 March 2025
DOI logo with link
https://doi.org/10.1075/jsls.00044.xu
Abstract
This study elicits 120 recordings as research data from the Test for English Majors Band 4 (TEM-4) Oral Test. We investigate the distinguishing cohesive devices across and between four pairs of speaking proficiency levels and the predictive cohesive devices for L2 speaking proficiency by examining all cohesion indices in TAACO for an independent speaking task. Results show that 15 local, 3 global, and 6 text cohesion indices distinguish across speaking proficiency levels. Besides, cohesion indices vary in differentiating powers at certain levels. In addition, 7 local, 5 global, and 1 text cohesion indices significantly correlate to L2 speaking proficiency levels. The regression model containing 2 local and 3 text cohesion indices explains 63.8% of the variance in predicting L2 speaking proficiency levels. These findings hold some implications for L2 speaking pedagogy and test assessment.
Article outline
  • 1.Introduction
  • 2.Cohesion and its relation to L2 quality/proficiency
    • 2.1Cohesion and cohesive devices
    • 2.2Cohesive devices in assessing L2 writing
    • 2.3Cohesion in oral production
  • 3.Methodology
    • 3.1Data
    • 3.2Instruments
    • 3.3Cohesion indices
      • 3.3.1Local cohesion indices
      • 3.3.2Global cohesion indices
      • 3.3.3Text cohesion indices
    • 3.4Data analysis
  • 4.Results
    • 4.1Distinctive cohesion indices across and between speaking proficiency levels
      • 4.1.1Local cohesion indices
      • 4.1.2Global cohesion indices
      • 4.1.3Text cohesion indices
    • 4.2Predictive indices for speaking proficiency levels
      • 4.2.1Correlations
      • 4.2.2Regression analysis: Training set
      • 4.2.3Regression model: Test set
  • 5.Discussion
  • 6.Conclusion
  • References
References (35)
References
Abdi Tabari, M., & Johnson, M. D. (2023). Exploring new insights into the role of cohesive devices in written academic genres. Assessing Writing, 571, 100749. Google Scholar logo with link to Google Scholar
Abdi Tabari, M., & Wind, A. M. (2023). Dynamic development of cohesive devices in English as a second language writing. International Review of Applied Linguistics in Language Teaching, 0(0). Google Scholar logo with link to Google Scholar
Chiang, S. Y. (1999). Assessing grammatical and textual features in L2 writing samples: The case of French as a foreign language. The Modern Language Journal, 83(2), 219–232. Google Scholar logo with link to Google Scholar
Covington, M. A., & McFall, J. D. (2010). Cutting the gordian knot: The moving-average type–token ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94–100. Google Scholar logo with link to Google Scholar
Crossley, S. A., & Kim, Y. (2019). Text integration and speaking proficiency: Linguistic, individual differences, and strategy use considerations. Language Assessment Quarterly, 16(2), 217–235. Google Scholar logo with link to Google Scholar
Crossley, S. A., & McNamara, D. S. (2010). Predicting second language writing proficiency: The roles of cohesion and linguistic sophistication. Journal of Research in Reading, 35(2), 115–135. Google Scholar logo with link to Google Scholar
Crossley, S. A., Kyle, K., & Dascalu, M. (2019). The tool for the automatic analysis of cohesion 2.0: Integrating semantic similarity and text overlap. Behavior Research Methods, 51(1), 14–27. Google Scholar logo with link to Google Scholar
Crossley, S. A., Kyle, K., & McNamara, D. S. (2016a). The development and use of cohesive devices in L2 writing and their relations to judgments of essay quality. Journal of Second Language Writing, 321, 1–16. Google Scholar logo with link to Google Scholar
(2016b). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48(4), 1227–1237. Google Scholar logo with link to Google Scholar
Crossley, S. A., & McNamara, D. S. (2013). Applications of text analysis tools for spoken response grading. Language Learning & Technology, 17(2), 171–192. Google Scholar logo with link to Google Scholar
Crossley, S. A., Clevinger, A., & Kim, Y. (2014). The role of lexical properties and cohesive devices in text integration and their effect on human ratings of speaking proficiency. Language Assessment Quarterly, 11(3), 250–270. Google Scholar logo with link to Google Scholar
Foster, P., Tonkyn, A., & Wigglesworth, G. (2000). Measuring spoken language: A unit for all reasons. Applied Linguistics, 211, 354–375. Google Scholar logo with link to Google Scholar
Gao, M., & Liu, Y. (2023). An analysis of the distinguishing features of discourse performance in an online diagnostic speaking test. Journal of China Examinations, (03), 60–67.Google Scholar logo with link to Google Scholar
Graesser, A. C., McNamara, D. S., & Louwerse, M. M. (2003). What do readers need to learn in order to process coherence relations in narrative and expository text. In A. P. Sweet & C. E. Snow (Eds.), Rethinking Reading Comprehension (pp. 82–98). New York, NY: Guilford Press.Google Scholar logo with link to Google Scholar
Gu, L., & Hsieh, C.-N. (2019). Distinguishing features of young english language learners’ oral performance. Language Assessment Quarterly, 16(2), 180–195. Google Scholar logo with link to Google Scholar
Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. Google Scholar logo with link to Google Scholar
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.Google Scholar logo with link to Google Scholar
(1985). Language, context and text: Aspects of language in social-semiotic perspective. Deakin University Press.Google Scholar logo with link to Google Scholar
Iwashita, N., May, L., & Moore, P. (2017). Features of discourse and lexical richness at different performance levels in the Aptis speaking test. ARAGs Research Reports, AR-G/2017/002. British Council. [URL]
Jin, Y., & Fan, J. (2011). Test for English majors (TEM) in China. Language Testing, 28(4), 589–596. Google Scholar logo with link to Google Scholar
Lee, C., Ge, H., & Chung, E. (2021). What linguistic features distinguish and predict L2 writing quality? A study of examination scripts written by adolescent Chinese learners of English in Hong Kong. System, 971, 102461. Google Scholar logo with link to Google Scholar
Lin, Y.-L., Tsai, Y.-S., & Hsieh, C.-Y. (2023). Discourse competence across band scores: An analysis of speaking performance in the general english proficiency test. International Review of Applied Linguistics in Language Teaching, 0(0). Google Scholar logo with link to Google Scholar
Liu, M., & Braine, G. (2005). Cohesive features in argumentative writing produced by Chinese undergraduates. System, 33(4), 623–636. Google Scholar logo with link to Google Scholar
Liu, Q., Zhou, S. (2024). Exploring Interaction Patterns in Web-Based Collaborative Writing Tasks Among Advanced L2 Learners: A Case Study Using Feishu Platform. In: Kubincová, Z., et al. Emerging Technologies for Education. SETE 2023. Lecture Notes in Computer Science, vol 14607. Springer, Singapore. Google Scholar logo with link to Google Scholar
Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal, 96(2), 190–208. Google Scholar logo with link to Google Scholar
McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. Google Scholar logo with link to Google Scholar
McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47(4), 292–330. Google Scholar logo with link to Google Scholar
Tyler, A. (1992). Discourse structure and the perception of incoherence in international teaching assistants’ spoken discourse. TESOL Quarterly, 26(4), 713. Google Scholar logo with link to Google Scholar
Tywoniw, R., & Crossley, S. (2019). The effect of cohesive features in integrated and independent L2 writing quality and text classification. Language Education and Assessment, 2(3), 110–134. Google Scholar logo with link to Google Scholar
Vögelin, C., Jansen, T., Keller, S. D., Machts, N., & Möller, J. (2019). The influence of lexical features on teacher judgements of ESL argumentative essays. Assessing Writing, 391, 50–63. Google Scholar logo with link to Google Scholar
Wang, T., & Zhou, D. (2023). Analyzing the distinguishing language features of Chinese EFL learners’ oral expression. Modern Foreign Languages, 46(06), 829–840.Google Scholar logo with link to Google Scholar
Witte, S. P., & Faigley, L. (1981). Coherence, cohesion, and writing quality. College Composition and Communication, 32(2), 189–204. Google Scholar logo with link to Google Scholar
Witten, I. H., Frank, E., & Hall, M. A. (2011). Data mining: Practical machine learning tools and techniques. Burlington, MA: Morgan Kaufmann.Google Scholar logo with link to Google Scholar
Yang, W., & Sun, Y. (2012). The use of cohesive devices in argumentative writing by Chinese EFL learners at different proficiency levels. Linguistics and Education, (231): 31–48. Google Scholar logo with link to Google Scholar
Yu, Hongliang. (2005). An exploratory study of narrative structure in Chinese tertiary EFL learners’ monologic production. Foreign Language Teaching and Research Press.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue