Cover not available

Article published In: TASK
Vol. 2:2 (2022) ► pp.184–217

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB
authorization required

Designing speaking tasks for different assessment goals

The complex relationship between cognitive task complexity, language performance, and task accomplishment

ORCiD logo with link | Waseda University, Japan
 | ETS Japan
Published online: 24 January 2023
DOI logo with link
https://doi.org/10.1075/task.21020.sas
Abstract
This study explored the role of cognitive task complexity in designing speaking assessment tasks. A sample of 120 English learners at different proficiency levels engaged in four narrative tasks with distinct levels/types of cognitive complexity. Performances were assessed using linguistic measures and a task accomplishment rubric. Findings revealed that one of the middle-complexity tasks with moderate intrinsic, moderate germane, and low extraneous complexity elicited the best linguistic performances overall, while the lowest complexity task elicited the highest task accomplishment ratings. Focusing on distinguishing among learner abilities, however, the highest complexity task demonstrated the best discrimination. Cognitive task complexity was also found to moderate the relationship between task accomplishment ratings and linguistic measures. Implications for research into task design, learner performance, and assessment purpose are considered.
Article outline
  • Introduction
  • Task-Based Language Assessment, task design, and cognitive task complexity
  • Task performance: Linguistic indices or task accomplishment?
  • Task design for different purposes
  • The current study
    • Methods
    • Participants
    • Materials
      • Narrative tasks
      • Cloze test
    • Procedures
    • Data scoring, coding, and analysis
      • Task accomplishment ratings
      • Linguistic indices
      • Syntactic complexity
      • Accuracy
      • Lexis
      • Fluency
      • Inter-coder reliability
      • Cloze test
      • Statistical analysis
  • Results
    • Proficiency test scores
    • Learner task performance
    • Correlations among performance measures and task accomplishment
    • Discrimination
  • Discussion
    • Role of task design in maximizing task performance
    • Role of task design in discriminating among proficiency levels
    • Role of task design in understanding the relationship among performance measures
  • Limitations
  • Conclusion
  • Acknowledgements
  • Note
  • References
References (48)
References
Brown, J. D. (1980). Relative merits of four methods for scoring cloze tests. The Modern Language Journal, 641, 311–317. Google Scholar logo with link to Google Scholar
(2005). Testing in language programs: a comprehensive guide to English language assessment. McGraw-Hill College.Google Scholar logo with link to Google Scholar
Brown, J. D., & Grüter, T. (2020). The same cloze for all occasions? Using the Brown (1980) cloze test for measuring proficiency in SLA research. International Review of Applied Linguistics. Google Scholar logo with link to Google Scholar
Brown, J. D., & Hudson, T. (1998). The alternatives in language assessment. TESOL quarterly, 32(4), 653–675. Google Scholar logo with link to Google Scholar
Brown, J. D., Hudson, T., Norris, J. M., & Bonk, W. (2002). Investigating task-based second language performance assessment. University of Hawai‘i Press.Google Scholar logo with link to Google Scholar
De Jong, N. H. (2018). Fluency in second language testing: Insights from different disciplines. Language Assessment Quarterly, 15(3), 237–254. Google Scholar logo with link to Google Scholar
De Jong, N. H., Steinel, M., Florijn, A. F., Schoonen, R., & Hulstijn, J. H. (2012). The effect of task complexity on functional adequacy, fluency and lexical diversity in speaking performances of native and non-native speakers. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA (pp. 121–142). John Benjamins. Google Scholar logo with link to Google Scholar
Doughty, C., & Pica, T. (1986). “Information gap” tasks: Do they facilitate second language acquisition? TESOL Quarterly, 20(2), 305–325. Google Scholar logo with link to Google Scholar
Educational Testing Service. (2019). TOEFL iBT® test speaking rubrics. [URL]
Freed, B. F., Segalowitz, N., & Dewey, D. P. (2004). Learning context and its effects on second language acquisition. Studies in Second Language Acquisition, 261, 275–301.Google Scholar logo with link to Google Scholar
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I. (2019). Multiple approaches to complexity in second language research. Second language research, 35(1), 3–21. Google Scholar logo with link to Google Scholar
Hunt, K. W. (1970). Syntactic maturity in school children and adults. Monographs of the Society for Research in Child Development, 351, 1–67. Google Scholar logo with link to Google Scholar
Koizumi, R. (2012). Relationships between text length and lexical diversity measures: Can we use short texts of less than 100 tokens? Vocabulary Learning and Instruction, 1(1), 60–69. Google Scholar logo with link to Google Scholar
Kuiken, F., & Vedder, I. (2017). Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 34(3), 321–336. Google Scholar logo with link to Google Scholar
Kuiken, F., Mos, M., & Vedder, I. (2005). Cognitive task complexity and second language writing performance. Eurosla Yearbook, 5(1), 195–222. Google Scholar logo with link to Google Scholar
Kuiken, F., Vedder, I., & Gilabert, R. (2010). Communicative adequacy and linguistic complexity in L2 writing. Communicative proficiency and linguistic development: Intersections between SLA and language testing research, EUROSLA Monograph Series 11, 81–100.Google Scholar logo with link to Google Scholar
Kyle, K., Crossley, S. A., & McNamara, D. S. (2016). Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing. Language Testing, 33(3), 319–340. Google Scholar logo with link to Google Scholar
Lee, J. (2020). Task closure and task complexity effects on L2 written performance. Journal of Second Language Writing, 501, 100777. Google Scholar logo with link to Google Scholar
Long, M. H., & Norris, J. M. (2000). Task-based language teaching and assessment. In M. Byram (Ed.), Encyclopedia of language teaching (pp. 597–603). Routledge.Google Scholar logo with link to Google Scholar
McNamara, T. (2000). Language testing. Oxford University Press.Google Scholar logo with link to Google Scholar
Norris, J. M. (2001). Identifying rating criteria for task-based EAP assessment. In T. D. Hudson & J. D. Brown (Eds.), A focus on language test development: Expanding the language proficiency construct across a variety of tests (pp. 163–204). University of Hawai‘i Press.Google Scholar logo with link to Google Scholar
(2010, September). Understanding instructed SLA: Constructs, contexts, and consequences. Plenary address delivered at the annual conference of the European Second Language Association (EUROSLA), Reggio Emilia, Italy.
(2015). Statistical significance testing in second language research: Basic problems and some solutions. In J. M. Norris, S. Ross & R. Schoonen (Eds.), Improving and extending quantitative reasoning in second language research (pp. 95–124). Wiley-Blackwell.Google Scholar logo with link to Google Scholar
(2016). Current uses for task-based language assessment. Annual Review of Applied Linguistics, 361, 230–244. Google Scholar logo with link to Google Scholar
Norris, J. M., & East, M. (2021). Task-based language assessment. In M. J. Ahmadian & M. Long (Eds.), The Cambridge Handbook of Task-Based Language Teaching. Cambridge University Press. Google Scholar logo with link to Google Scholar
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in Instructed SLA: The case of complexity. Applied Linguistics, 30(4), 555–578. Google Scholar logo with link to Google Scholar
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based L2 performance assessment. Language Testing, 19(4), 395–418. Google Scholar logo with link to Google Scholar
Pallotti, G. (2009). CAF: Defining, refining and differentiating constructs. Applied Linguistics, 30(4), 590–601. Google Scholar logo with link to Google Scholar
Révész, A. (2014). Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics, 351, 87–92. Google Scholar logo with link to Google Scholar
Révész, A., Ekiert, M., & Torgersen, E. N. (2016a). The effects of complexity, accuracy, and fluency on communicative adequacy in oral task performance. Applied Linguistics, 37(6), 828–848.Google Scholar logo with link to Google Scholar
Révész, A., Michel, M., & Gilabert, R. (2016b). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgments: A validation study. Studies in Second Language Acquisition, 38(4), 703–737. Google Scholar logo with link to Google Scholar
Robinson, P. (2001a). Task complexity, cognitive resources, and syllabus design: a triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 287–318). Cambridge University Press. Google Scholar logo with link to Google Scholar
(2001b). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 221, 27–57. Google Scholar logo with link to Google Scholar
Robinson, P., & Ross, S. (1996). The development of task-based assessment in English for academic purposes programs. Applied Linguistics, 17(4), 455–476. Google Scholar logo with link to Google Scholar
Sasayama, S. (2015). Validating the assumed relationship between task design, cognitive complexity, and second language task performance. Unpublished doctoral dissertation, Georgetown University.
(2016). Is a ‘complex’ task really complex? Validating the assumption of cognitive task complexity. The Modern Language Journal, 100(1), 231–254. Google Scholar logo with link to Google Scholar
Sasayama, S., & Norris, J. M. (2019). Unravelling cognitive task complexity: Learning from learners’ perspectives on task characteristics and L2 performance. In Z. Wen & M. J. Ahmadian (Eds.), Researching L2 task performance and pedagogy: In honour of Peter Skehan (pp. 95–132). John Benjamins. Google Scholar logo with link to Google Scholar
Sasayama, S., Garcia Gomez, P., & Norris, J. M. (2021). Designing efficient L2 writing assessment tasks for low-proficiency learners of English. TOEFL Research Report. Princeton, NJ: Educational Testing Service. Google Scholar logo with link to Google Scholar
Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press.Google Scholar logo with link to Google Scholar
(2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 301, 510–532. Google Scholar logo with link to Google Scholar
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 871, 245–251. Google Scholar logo with link to Google Scholar
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 41, 295–312. Google Scholar logo with link to Google Scholar
Treffers-Daller, J., Parslow, P., & Williams, S. (2018). Back to basics: How measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics, 39(3), 302–327.Google Scholar logo with link to Google Scholar
Van Gorp, K., & Deygers, B. (2014). Task-based language assessment. In A. Kunnan (Ed.), The companion to language assessment, 21 (pp. 578–593). John Wiley & Sons.Google Scholar logo with link to Google Scholar
Vasylets, O., Gilabert, R., & Manchon, R. M. (2020). Task modality, communicative adequacy and CAF measures: The moderating role of task complexity. In Manchon, R. M. (Ed.), Writing and language learning: Advancing research agendas (pp. 183–206). John Benjamins. Google Scholar logo with link to Google Scholar
Xi, X., & Norris, J. M. (Eds.). (2021). Assessing academic English for higher education admissions. Routledge. Google Scholar logo with link to Google Scholar
Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31(2), 236–259. Google Scholar logo with link to Google Scholar
Cited by (4)

Cited by four other publications

Chapman, Mark, Meg Montee, Yangting Wang, Gordon Blaine West, Jason A. Kemp & Ahyoung Alicia Kim
2025. Speaking task design and complexity and fluency features of adolescent English learners. Language Testing DOI logo
O'Grady, Stefan
2025. Task design and rater effects in task‐based language assessment. TESOL Journal 16:1 DOI logo
Norris, John M., Shoko Sasayama & Michelle Kim
2023. Simulating Real‐World Context in an Email Writing Task: Implications for Task‐Based Language Assessment. ETS Research Report Series 2023:1 ► pp. 1 ff. DOI logo
[no author supplied]
2025. Second Language Cognitive Task Complexity [Task-Based Language Teaching, 18], DOI logo

This list is based on CrossRef data as of 5 december 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue