Cover not available

Article published In: Journal of Second Language Studies
Vol. 6:1 (2023) ► pp.95–118

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
EPUB logo with download linkDownload EPUB

Pitfalls of production data analysis for investigating L2 cognitive mechanism

An ontological realism perspective

 | Chuo University
 | Matsuyama University
 | Kansai University
Open Access logo
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.

For any use beyond this license, please contact the publisher at rights@benjamins.nl.

Published online: 22 March 2022
DOI logo with link
https://doi.org/10.1075/jsls.21013.fuk
Abstract
This article addresses the pitfalls of performance analysis in investigating cognitive processing during second language (L2) learning. The problems that we discuss in this paper are twofold: (1) Assuming psychological variables to be ontological entities without meeting the criteria for ontological reality and (2) Inappropriateness of assessing abilities based on learner’s speaking or writing performance to investigate cognitive processes. By addressing these problems, we argue that some latent variables postulated by observing L2 performance do not exist in reality and emphasize the difficulty of interpreting cognitive mechanisms through performance analysis. We also enumerate some problems that arise from the epistemological perspectives of previous research practice (e.g., the bifurcation of contradictory hypotheses and their indeterminacy). Finally, two alternative approaches treating L2 performance are proposed. The implications of this line of discussion for future research are also discussed.
Article outline
  • 1.Introduction
  • 2.Ontological perspective on performance analysis
  • 3.Skepticism of CAF as ontological entities
  • 4.L2 performance as superficial phenomena that form a complex structure
  • 5.Implications for future research
  • Notes
  • References
References (83)
References
Biber, D., Gray, B., & Poonpon, K. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly, 31(1), 5–35. Google Scholar logo with link to Google Scholar
Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110(2), 305–314. Google Scholar logo with link to Google Scholar
Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 91, 91–121. Google Scholar logo with link to Google Scholar
Borsboom, D., Cramer, A. O. J., Kievit, R. A., Zand Scholten, A., & Franić, S. (2009). The end of construct validity. In R. W. Lissitz (Ed.), The concept of validity (pp. 135–170). Charlotte, NC: Information Age Publishers.Google Scholar logo with link to Google Scholar
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. Google Scholar logo with link to Google Scholar
Branigan, H. P., & Pickering, M. J. (2017). An experimental approach to linguistic representation. Behavioral and Brain Sciences, 401, e282. Google Scholar logo with link to Google Scholar
Bulté, B., & Housen, A. (2018). Syntactic complexity in L2 writing: Individual pathways and emerging group trends. International Journal of Applied Linguistics, 28(1), 147–164. Google Scholar logo with link to Google Scholar
Cartwright, N. (1983). How the laws of physics lie. Oxford: Clarendon Press. Google Scholar logo with link to Google Scholar
Churchland, P. (1995). The engine of reason, the seat of the soul: A philosophical journey into the brain. Cambridge, MA: MIT Press. Google Scholar logo with link to Google Scholar
Coltman, T., Devinney, T. M., Midgley, D. F., & Venaik, S. (2008). Formative versus reflective measurement models: Two applications of formative measurement. Journal of Business Research, 61(12), 1250–1262. Google Scholar logo with link to Google Scholar
Declerck, M., & Kormos, J. (2012). The effect of dual task demands and proficiency on second language speech production. Bilingualism: Language and Cognition, 15(4), 782–796. Google Scholar logo with link to Google Scholar
Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5(2), 155–174. Google Scholar logo with link to Google Scholar
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
Fornell, C., & Bookstein, F. L. (1982). Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research, 19(4), 440–452. Google Scholar logo with link to Google Scholar
Foster, P., & Tavakoli, P. (2009). Native speakers and task performance: Comparing effects on complexity, fluency, and lexical diversity. Language learning, 59(4), 866–896. Google Scholar logo with link to Google Scholar
Fukuta, J., Goto, A., Kawaguchi, Y., Kurita, A., & Murota, D. (2018). Syntactically–driven algorithmic processing of PP-attachment ambiguity in a second language. International Review of Applied Linguistics in Language Teaching, 56(3), 253–277. Google Scholar logo with link to Google Scholar
Fukuta, J., Goto, A., Kawaguchi, Y., Murota, D., & Kurita, A. (2015). Japanese EFL learners’ implicit knowledge and algorithmic processing of dative alternation: Perspective from syntactic priming in reading comprehension. ARELE: Annual Review of English Language Education in Japan, 261, 221–236. Google Scholar logo with link to Google Scholar
Fukuta, J., & Yamashita, J. (2015). Effects of cognitive demands on attention orientation in L2 oral production. System, 531, 1–12. Google Scholar logo with link to Google Scholar
García Mayo, M. D. P. (Ed.). (2006). Investigating tasks in formal language learning. Multilingual Matters. Google Scholar logo with link to Google Scholar
Gilabert, R. (2007a). The simultaneous manipulation of task complexity along planning time and (+ / - here-and-now): Effects on L2 oral production. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 44–68). Clevedon, UK: Multilingual Matters.Google Scholar logo with link to Google Scholar
(2007b). Effects of manipulating task complexity on self-repairs during L2 oral production, International Review of Applied Linguistics in Language Teaching, 45(3), 215–240. Google Scholar logo with link to Google Scholar
Gilabert, R., Barón, J., & Llanes, À. (2009). Manipulating cognitive complexity across task types and its impact on learners’ interaction during oral performance. International Review of Applied Linguistics in Language Teaching, 47(3), 367–395. Google Scholar logo with link to Google Scholar
Hacking, I. (1983). Representing and intervening: Introductory topics in the philosophy of science. Cambridge: Cambridge University Press. Google Scholar logo with link to Google Scholar
Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E. (2004). Is Syntax Separate or Shared Between Languages?: Cross-Linguistic Syntactic Priming in Spanish-English Bilinguals. Psychological Science, 15(6), 409–414. Google Scholar logo with link to Google Scholar
Hiver, P., & Al-hoorie, A. H. (2016). A dynamic ensemble for second language research: Putting complexity theory into practice. The Modern Language Journal, 100(4), 741–756. Google Scholar logo with link to Google Scholar
Housen, A., & Kuiken, F. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30(4), 461–473. Google Scholar logo with link to Google Scholar
Inhelder, B., & Piaget, J. (1958). An essay on the construction of formal operational structures. the growth of logical thinking: From childhood to adolescence. A. Parsons & S. Milgram (Trans.). New York: Basic Books. Google Scholar logo with link to Google Scholar
Ishikawa, T. (2007). The effect of manipulating task complexity along the [+ / - here-and-now] dimension on L2 written narrative discourse. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 136–156). Clevedon, UK: Multilingual Matters.Google Scholar logo with link to Google Scholar
(2008). The effect of task demands of intentional reasoning on L2 speech performance. The Journal of Asia TEFL, 51, 29–63.Google Scholar logo with link to Google Scholar
Izumi, E., Uchimoto, K., & Isahara, H. (2004). The NICT-JLE Corpus: Exploiting the language learners’ speech database for research and education. International Journal of the Computer, the Internet and Management, 12(2), 119–125.Google Scholar logo with link to Google Scholar
Jackson, D. O., & Suethanapornkul, S. (2013). The Cognition Hypothesis: A synthesis and meta-analysis of research on second language task complexity. Language Learning, 63(2), 330–367. Google Scholar logo with link to Google Scholar
Jolliffe, I. T. (2002). Principal component analysis (2nd ed). New York: Springer.Google Scholar logo with link to Google Scholar
Kaan, E., & Chun, E. (2018). Priming and adaptation in native speakers and second-language learners. Bilingualism: Language and Cognition, 21(2), 228–242. Google Scholar logo with link to Google Scholar
Kaan, E., Futch, C., Fuertes, R. F., Mujcinovic, S., & Fuente, E. Á. D. L. (2019). Adaptation to syntactic structures in native and nonnative sentence comprehension. Applied Psycholinguistics, 40(1), 3–27. Google Scholar logo with link to Google Scholar
Kim, Y. (2009). The effects of task complexity on learner-learner interaction. System, 371, 254–268. Google Scholar logo with link to Google Scholar
Koizumi, R., & In’nami, Y. (2014). Modeling complexity, accuracy, and fluency of Japanese Learners of English: A structural equation modeling approach. JALT Journal, 36(1), 25–46. Google Scholar logo with link to Google Scholar
Kormos, J., & Trebits, A. (2012). The role of task complexity, modality, and aptitude in narrative task performance. Language Learning, 621, 439–472. Google Scholar logo with link to Google Scholar
Kuiken, F., & Vedder, I. (2007). Task complexity and measures of linguistic performance in L2 writing. International Review of Applied Linguistics in Language Teaching, 451, 261–284. Google Scholar logo with link to Google Scholar
(2008). Cognitive task complexity and written output in Italian and French as a foreign language. Journal of Second Language Writing, 171, 48–60. Google Scholar logo with link to Google Scholar
(2012). Syntactic complexity, lexical variation and accuracy as a function of task complexity and proficiency level in L2 writing and speaking. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy, and fluency in SLA (pp. 143–170). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Lambert, C., & Kormos, J. (2014). Complexity, accuracy, and fluency in task-based L2 research: Toward more developmentally based measures of second language acquisition. Applied Linguistics, 35(5), 607–614. Google Scholar logo with link to Google Scholar
Lambert, C., Kormos, J., & Minn, D. (2017). Task repetition and second language speech processing. Studies in Second Language Acquisition, 39(1), 167–196. Google Scholar logo with link to Google Scholar
Larsen-Freeman, D. (2006). The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English. Applied Linguistics, 27(4), 590–619. Google Scholar logo with link to Google Scholar
Laudan, L. (1981). A confutation of convergent realism. Philosophy of Science, 48(1), 19–49. Google Scholar logo with link to Google Scholar
Markus, K. A., & Borsboom, D. (2013). Frontiers of test validity theory: Measurement, causation, and meaning. Abingdon, UK: Routledge. Google Scholar logo with link to Google Scholar
Mehnert, U. (1998). The effects of different lengths of time for planning on second language performance. Studies in Second Language Acquisition, 20(1), 83–108. Google Scholar logo with link to Google Scholar
Nishimura, Y., Kawaguchi, Y., & Abe, D. (2018). The relationship between complexity and fluency in L2 writing: An approach using network analysis. Language Education & Technology, (55), 171–198.Google Scholar logo with link to Google Scholar
Nishimura, Y., Tamura, Y., & Fukuta, J. (2018). Network structures in L2 oral performance: A learner corpus study. ARELE: Annual Review of English Language Education in Japan, 291, 113–128. Google Scholar logo with link to Google Scholar
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30(4), 555–578. Google Scholar logo with link to Google Scholar
Ong, J., & Zhang, L. J. (2010). Effects of task complexity on the fluency and lexical complexity in EFL students’ argumentative writing. Journal of Second Language Writing, 19(4), 218–233. Google Scholar logo with link to Google Scholar
Pallotti, G. (2009). CAF: Defining, refining, and differentiating constructs. Applied Linguistics, 30(4), 590–601. Google Scholar logo with link to Google Scholar
(2015). A simple view of linguistic complexity. Second Language Research, 31(1), 117–134. Google Scholar logo with link to Google Scholar
Pickering, M. J., & Branigan, H. P. (1999). Syntactic priming in language production. Trends in cognitive sciences, 3(4), 136–141. Google Scholar logo with link to Google Scholar
Plonsky, L., & Kim, Y. (2016). Task-Based Learner Production: A Substantive and Methodological Review. Annual Review of Applied Linguistics, 361, 73–97. Google Scholar logo with link to Google Scholar
Polat, B., & Kim, Y. (2014). Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35(2), 184–207. Google Scholar logo with link to Google Scholar
Révész, A. (2014). Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics, 35(1), 87–92. Google Scholar logo with link to Google Scholar
Reynolds, C. W. (1987). Flocks, herds and schools: A distributed behavioral model. SIGGRAPH ’87: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing Machinery. pp. 25–34.
Robinson, P. (1995). Task complexity and second language narrative discourse. Language Learning, 45(1), 99–140. Google Scholar logo with link to Google Scholar
(2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22(1), 27–57. Google Scholar logo with link to Google Scholar
(2003). Attention and memory during SLA. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 631–678). Malden, MA: Blackwell. Google Scholar logo with link to Google Scholar
(2007). Criteria for classifying and sequencing pedagogic tasks. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 7–26). Clevedon, UK: Multilingual Matters.Google Scholar logo with link to Google Scholar
Robinson, P. J. (2011). Second language task complexity, the cognition hypothesis, language learning, and performance. In P. J. Robinson (Ed.), Second language task complexity: Researching the cognition hypothesis of language learning and performance (pp. 2–38). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Romeijn, J. W. (2008). The all-too-flexible abductive method: ATOM’s normative status. Journal of Clinical Psychology, 64(9), 1023–1036. Google Scholar logo with link to Google Scholar
Schmittmann, V. D., Cramer, A. O. J., Waldorp, L. J., Epskamp, S., Kievit, R. A., & Borsboom, D. (2013). Deconstructing the construct: A network perspective on psychological phenomena. New Ideas in Psychology, 31(1), 43–53. Google Scholar logo with link to Google Scholar
Sheppard, C. (2004). The measurement of second language production: The validity of fluency, accuracy, and complexity. ICU Language Research Bulletin, 191, 139–156.Google Scholar logo with link to Google Scholar
Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied Linguistics, 17(1), 38–62. Google Scholar logo with link to Google Scholar
(1998). A cognitive approach to language learning. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
(2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30(4), 510–532. Google Scholar logo with link to Google Scholar
(2014). The context researching a processing perspective on task performance. In P. Skehan (Ed.), Processing perspectives on task performance (pp. 1–26). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Skehan, P., & Foster, P. (1997). Task type and task processing conditions as influences on foreign language performance. Language Teaching Research, 1(3), 185–211. Google Scholar logo with link to Google Scholar
(1999). The influence of task structure and processing conditions on narrative retellings. Language learning, 49(1), 93–120. Google Scholar logo with link to Google Scholar
(2005). Strategic and on-line planning: The influence of surprise information and task time on second language performance. In R. Ellis (Ed.), Planning and task performance in a second language (pp. 193–216). Amsterdam: John Benjamins. Google Scholar logo with link to Google Scholar
Spoelman, M., & Verspoor, M. (2010). Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics, 31(4), 532–553. Google Scholar logo with link to Google Scholar
Tavakoli, P., & Foster, P. (2011). Task design and second language performance: The effect of narrative type on learner output. Language Learning, 611, 37–72. Google Scholar logo with link to Google Scholar
Van Fraassen, B. (1980). The scientific image. Oxford: Oxford University Press. Google Scholar logo with link to Google Scholar
Vercellotti, M. L. (2017). The development of complexity, accuracy, and fluency in second language performance: A longitudinal study. Applied Linguistics, 38(1), 90–111. Google Scholar logo with link to Google Scholar
Wolfe-Quintero, K., Inagaki, S., & Kim, H.-Y. (1998). Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu University of Hawai’i, Second Language Teaching and Curriculum Center.Google Scholar logo with link to Google Scholar
Wright, C. (2020). Effects of task type on L2 Mandarin fluency development. Journal of Second Language Studies, 3(2), 157–179. Google Scholar logo with link to Google Scholar
Xu, T. S., Zhang, L. J., & Gaffney, J. S. (2021). Examining the relative effectiveness of task complexity and cognitive demands on students’ writing in a second language. Studies in Second Language Acquisition. Advance online publication. Google Scholar logo with link to Google Scholar
Yu, H., & Lowie, W. (2019). Dynamic paths of complexity and accuracy in second language speech: A longitudinal case study of Chinese learners. Applied Linguistics, 41(6), 855–877. Google Scholar logo with link to Google Scholar
Yuan, F., & Ellis, R. (2003). The effects of pre-task planning and on-line planning on fluency, complexity and accuracy in L2 monologic oral production. Applied Linguistics, 24(1), 1–27. Google Scholar logo with link to Google Scholar
Cited by (4)

Cited by four other publications

Alamer, Abdullah
2025. Structural equation modeling (SEM) in L2 writing research: Simple tutorial and useful recommendations. Research Methods in Applied Linguistics 4:2 ► pp. 100202 ff. DOI logo
Vitta, Joseph P., Aaron Hahn, Derek Canning, Daniel R. Isbell, Ali H. Al-Hoorie & Christopher Nicklin
2025. Conceptualization and frequency of sampling (mal)practices in L2 inferential quantitative research. Journal of Second Language Studies DOI logo
Alamer, Abdullah, Florian Schuberth & Jörg Henseler
2024. When and how to use confirmatory composite analysis (CCA) in second language research. Studies in Second Language Acquisition 46:2 ► pp. 597 ff. DOI logo
Wagner, Thomas
2023. Implications for the Classroom. In English Interlanguage Morphology, ► pp. 113 ff. DOI logo

This list is based on CrossRef data as of 13 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.

Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue