Cover not available

Article published In: ITL - International Journal of Applied Linguistics
Vol. 127/128 (2000) ► pp.1–36

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload PDF
authorization required

Tests as a Second Language Research Method

Their Types, Reliability, Validity, and Variable Research Results

 | Aichi Gakuin University (Japan)
Published online: 1 January 2000
DOI logo with link
https://doi.org/10.1075/itl.127-128.01ito
Abstract
In second language acquisition (SLA) research various tests are employed for theory building. However, what tests should be administered to participants in a research setting? In order to address the issue, this study was conducted to examine the reliability, validity of tests themselves, and the difference in results according to the difference of test-types, focusing on cross-sectional restrictive relative clause acquisition and relative clause test-types. Among the variety of relative clause tests, four test-types that appear frequently in recent SLA academic publications were selected. The four test-types were Translation, Cloze Procedure, Grammaticality Judgment, and Sentence Combining. The results from 120 Japanese students indicate the following. First, Sentence Combining shows high reliability in internal consistency and the highest validity. Second, the research results change across the different test-types. Third, the concept of 'test-type (task) related interlanguage variability' should be explained by the combination of quality-related issues such as 'measurement error' and the cognitive demands each test-type requires of the subjects. Implications for the practical issues of language test construction for SLA research, educational evaluation, and the directions for further research are also discussed.
References (98)
References
AARTS, F. & SCHILS, E. (1995) : Relative clauses, the accessibility hierarchy and the contrastive analysis hypothesis. Internationa Review of Applied Linguistics 331, 47–63.Google Scholar logo with link to Google Scholar
AKAGAWA, Y. (1992) : Avoidance of relative clauses by Japanese high school students. JACET Bulletin 231, 1–18.Google Scholar logo with link to Google Scholar
ALDERSON, J.C., CLAPHAM, C, WALL, D. (1995) : Language test construction and evaluation. Cambridge: Cambridge University Press.Google Scholar logo with link to Google Scholar
ARTHUR, B. (1980) : Gauging the boundaries of second language competence : A study of learner judgment. Language Learning 301, 178–194. Google Scholar logo with link to Google Scholar
BACHMAN, L.F. (1990) : Fundamental considerations in language testing. Oxford: Oxford University Press.Google Scholar logo with link to Google Scholar
BAILEY, N., EISENSTEIN, M. & MADDEN, C. (1976) : The development of wh- questions in adult second language learners. In Fanslow, J. & Crymes, R. (Eds.), ON TESOL '761, 1-9. Washington, D.C: TESOL.Google Scholar logo with link to Google Scholar
BEEBE, L.M. (1980) : Sociolinguistic variation and style shifting in second Language acquisition. Language Learning 301, 178–194. Google Scholar logo with link to Google Scholar
(1988) : Five sociolinguistic approaches to second language acquisition. In Beebe, L.M. (Ed.), Issues in second language acquisition : Multiple perspectives (pp. 43–77). New York : Newbury House.Google Scholar logo with link to Google Scholar
BROWN, J.D. (1988) : Understanding research in second language learning : A teacher's guide to statistics and research design. London : Cambridge University Press.Google Scholar logo with link to Google Scholar
(1996) : Testing in language programs. Upper Saddle River, NJ : Prentice Hall Regents.Google Scholar logo with link to Google Scholar
BUCK, G. (1992) : Translation as a language testing procedure : Does it work? Language Testing 91, 123–148. Google Scholar logo with link to Google Scholar
CELCE-MURCIA, M. & LARSEN-FREEMAN, D.E. (1983) : The grammar book-An ESL/EFL teacher's course. Rowley, MA : Newbury House.Google Scholar logo with link to Google Scholar
CHAUDRON, C. (1983) : Research on metalinguistic judgments : A review of theory, methods, and results. Language Learning 331, 343–377. Google Scholar logo with link to Google Scholar
DEKEYSER, R. (1990) : Towards a valid measurement of monitored knowledge. Language Testing 71, 147–157. Google Scholar logo with link to Google Scholar
DlCKERSON, L.J. (1975) : The learner's interlanguage as a system of variable rules. TESOL Quarterly 91, 401–407. Google Scholar logo with link to Google Scholar
DUFF, A. P. (1993) : Tasks and interlanguage : An SLA perspective. In Crookes, G. & Gass, S.M. (Eds.), Tasks and language learning : Integrating theory & practice (pp. 57–95). Bristol, PA : Multilingual Matters.Google Scholar logo with link to Google Scholar
DULAY, H., BURT, M. & KRASHEN, S.D. (1982) : Language two. Oxford : Oxford University Press.Google Scholar logo with link to Google Scholar
ECKMAN, F.R., BELL, L. & NELSON, D. (1988) : On the generalization of relative clause instruction in the acquisition of English as a second language. Applied Linguistics 91, 1–20. Google Scholar logo with link to Google Scholar
EBEL, R.L. (1979) : Essentials of educational measurement (Third edition.). Prentice-Hall, Inc.Google Scholar logo with link to Google Scholar
EISENSTEIN, M., BAILEY, N. & MADDEN, C. (1982) : It takes two : Contrasting tasks and contrasting structures. TESOL Quarterly 161, 381–391. Google Scholar logo with link to Google Scholar
ELLIS, R. (1986) : Sources of variability in interlanguage. Applied Linguistics 61, 118–131. Google Scholar logo with link to Google Scholar
(1991) : Grammar judgments and second language acquisition. Studies in Second Language Acquisition 131, 161–186. Google Scholar logo with link to Google Scholar
(1994). Data, theory, and applications in second language acquisition research. In Ellis, R. The study of second language acquisition (pp. 669–691). Oxford : Oxford University Press.Google Scholar logo with link to Google Scholar
FULCHER, G. (1995) : Variable competence in second language acquisition : A problem for research methodology? System 231, 25–33. Google Scholar logo with link to Google Scholar
GASS, S. (1979) : Language transfer and universal grammatical relations. Language Learning 291, 327–344. Google Scholar logo with link to Google Scholar
(1980) : An investigation of systematic transfer in adult second language learners. In Scarcella, R.C., & Krashen, S.D. (Eds.), Research in second language acquisition (pp. 132–141). Rowley, MA : Newbury House.Google Scholar logo with link to Google Scholar
(1982) : From theory to practice. In Hines, M. & Rutherford, W. (Eds.), On TESOL '81. Washington, D.C. : TESOL, 129–139.Google Scholar logo with link to Google Scholar
(1983) : The development of L2 intuitions. TESOL Quarterly 171, 273–291. Google Scholar logo with link to Google Scholar
GASS, S. & SELINKER, L. (1994) : Second language acquisition : An introductory course. Hillsdale, NJ : Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
HATCH, E. & LAZARTON, A. (1991) : The research manual. New York : Newbury House.Google Scholar logo with link to Google Scholar
HENNING, G. (1987) : A guide to language testing : Development, evaluation, research. Boston, MA : Heinle & Heinle Publishers.Google Scholar logo with link to Google Scholar
HUEBER, T. (1985) : System and variability in interlanguage syntax. Language Learning 351, 141–163. Google Scholar logo with link to Google Scholar
HULL, J. (1986). Task variation in interlanguage performance : Does it Affect monitoring? Working Papers 5. Department of ESL. The University of Hawaii at Manoa.Google Scholar logo with link to Google Scholar
HULSTIJN, J. (1989): A cognitive view on interlanguage variability. In Eisenstein (Ed.), The dynamic interlanguage : Empirical studies in second language variation (pp. 17–31). New York : Plenum Press.Google Scholar logo with link to Google Scholar
HYLTENSTAM, K. (1983) : Data type and second language variability. In Rongbon, H. (ed.), Psycholinguistics and foreign language learning (pp. 57–74). Abo : Abo Akademi.Google Scholar logo with link to Google Scholar
IKEBE, Y. (1990) : A study on the acquisition of relative clauses by Japanese learners of English. Unpublished MA thesis, Hiroshima University.Google Scholar logo with link to Google Scholar
INOI, S. (1991) : Variation in interlanguage with special reference to articles and pronouns. Annual Review of English Language Education in Japan 21, 1–10.Google Scholar logo with link to Google Scholar
ITO, A. (1995) : A study on the effects of difference of test-types on interlanguage performance of Japanese EFL learners (Manuscript). A paper presented at 26th CASELE Conference at Yamaguchi University, Japan.Google Scholar logo with link to Google Scholar
(1996a) : Testing English tests : A language proficiency perspective. JALT Journal181, 183–197.Google Scholar logo with link to Google Scholar
(1996b) : A study on the variability of test performance of Japanese EFL learners : A combination of two theoretical frameworks. CELES Bulletin 261, 227–234.Google Scholar logo with link to Google Scholar
(1997): Japanese EFL learners' test-type related interlanguage variability. JALT Journal 191, 89–105.Google Scholar logo with link to Google Scholar
(1998) : The author responds : More on Test-Type. JALT Journal 201, 89–90.Google Scholar logo with link to Google Scholar
(1999) : A study of test-type related variability of interlanguage performance among Japanese EFL learners : A focus on relative clause tests. Unpublished Ph.D. Dissertation, Hiroshima University.Google Scholar logo with link to Google Scholar
JONES, A. (1991) : Generalization in the acquisition of relative clauses in English. Annual Report of Studies : The Faculty of Letters of Jissen Women's University 331, 1–39.Google Scholar logo with link to Google Scholar
KAMEYAMA, T. (1987) : Bunpo test niokeru test keishiki niyoru keitaiso seitoritsu no henka. (The change of accuracy rate in grammar tests by the difference of test-types). CELES Bulletin 171, 248–252.Google Scholar logo with link to Google Scholar
KAWAUCHI, C. (1988) : Universal processing of relative clauses by adult learners of English. JACET Bulletin 191, 19–36.Google Scholar logo with link to Google Scholar
KEENAN, E.L. (1975) : Variation in universal grammar. In Fasold, R. & Shuy, R. (Eds.), Analyzing variation in language (pp. 136–148). Washington, D.C. : Georgetown University Press.Google Scholar logo with link to Google Scholar
KEENAN, E.L., & COMRIE, B. (1977) : Noun phrase accessibility and universal grammar. Linguistic Inquiry 81, 63–99.Google Scholar logo with link to Google Scholar
KLEIN-BRALEY, C. (1982) : Die Ubersetzung als Testverfahren in der Staatprufung fur Lerramtskandidaten. Neusprachliche Mitteilungen 21, 94–97.Google Scholar logo with link to Google Scholar
(1987) : Fossil at large : Translation as a language testing procedure. Colloquium on language testing research, April, 6–9.Google Scholar logo with link to Google Scholar
KRASHEN, S.D. (1977a) : The monitor model for adult second language performance. In Burt, M., Dulay, H. & Finoccjiaro, M., Viewpoints on English as a second language (pp. 152–161). New York : Regents.Google Scholar logo with link to Google Scholar
(1977b) : Some issues relating to the monitor model. In Brown, D.H., Yorio, CA., & Crymes, R.H. (Eds.), ON TESOL '771, 144-158. Washington, D.C. : TESOL.Google Scholar logo with link to Google Scholar
(1978a) : Individual variation in the use of the monitor. In Ritchie, W.C. (Ed.), Second language acquisition research : Issues and implications (pp. 175–183). New York : Academic Press.Google Scholar logo with link to Google Scholar
(1978b) : The monitor model for second language acquisition. In Gingras, R.C. (Ed.), Second language acquisition and foreign language teaching (pp. 1–26). New York : Center for Applied Linguistics.Google Scholar logo with link to Google Scholar
(1981) : Second language acquisition and second language learning. Oxford : Pergamon Press.Google Scholar logo with link to Google Scholar
(1982) : Principles and practice in second language acquisition. Oxford : Pergamon Press.Google Scholar logo with link to Google Scholar
(1985) : The input hypothesis : Issues and implications. London : Longman.Google Scholar logo with link to Google Scholar
KRASHEN, S.D. & TERRELL, T. (1983) : The natural approach : Language acquisition in the classroom. Oxford : Pergamon Press.Google Scholar logo with link to Google Scholar
KUBOTA, M. (1993) : Accuracy order and frequency order of relative clauses as used by Japanese senior high school students of EFL. IRLT Bulletin 71, 27–53.Google Scholar logo with link to Google Scholar
KURODA, T. (1986) : A research on cross-language influence in Japanese EFL learners. Unpublished MA thesis, Hyogo University of Teacher Education.Google Scholar logo with link to Google Scholar
KUNO, S. (1976) : Subject, theme, and the speaker's empathy - a reexamination of relativization phenomena. In Li, C. (Ed.), Subject and topic (pp. 417–444). Academic Press.Google Scholar logo with link to Google Scholar
LEE, Y.O., KRASHEN, S.D., and GRIBBONS, B. (1995) : The effects of reading on the acquisition of English relative clauses. I.T.L. Review of Applied Linguistics113–114, 263–273.Google Scholar logo with link to Google Scholar
LARSEN-FREEMAN, D.E. (1975): The acquisition of grammatical morphemes by adult ESL students. TESOL Quarterly 91, 409–419. Google Scholar logo with link to Google Scholar
LARSEN-FREEMAN, D. LONG, M.H. (1991) : An introduction to second language acquisition research. London : Longman.Google Scholar logo with link to Google Scholar
LOSCHKY, L. & BLEY-VROMAN, R. (1993) : Grammar and task-based methodology. In Crookes, G. & Gass, S.M. (Eds.), Tasks and language learning : Integrating theory and practice (pp. 123–167). Bristol, PA : Multilingual Matters.Google Scholar logo with link to Google Scholar
LONG, M.H. & SATO, C.J. (1984) : Methodological issues in interlanguage studies : An interactionist perspective. In A. Davies, A., C. Criper., & A. P. R. Howatt, (Eds.), Interlanguage (pp. 253–279). Edinburgh : Edinburgh University Press.Google Scholar logo with link to Google Scholar
MAKONI, S.B. (1996) : Variation in unplanned discourse. International Review of Applied Linguistics 341, 167–181. Google Scholar logo with link to Google Scholar
MOCHIZUKI, A. (1996) : A comparison of C-tests with four different texts : Their reliability and validity. A paper presented at 22nd FELES Conference at Tohoku Gakuin Unviersity, Japan.Google Scholar logo with link to Google Scholar
OHBA, H. (1987) : Task variation riron ni tsuite. (On the task variation theory). CELES Bulletin 171, 43–49.Google Scholar logo with link to Google Scholar
(1994a): Nihonjin eigo gakushusha no chukangengo kahen-sei: Task keishiki no kanten kara. (Japanese EFL learners' interlanguage variability : With reference to the effects of task-type) CELES Bulletin 241, 187–192.Google Scholar logo with link to Google Scholar
(1994b): Test keishiki no chigai niyoru eigo gakushusha no performance no kahensei. (Task-related variability in interlanguage by Japanese learners of English) STEP Bulletin 61, 34–48.Google Scholar logo with link to Google Scholar
(1995) : The learning order of English relative clauses by Japanese senior high school students in an instruction-only environment. Journal of Health Sciences University of Hokkaido, 211, 19–35.Google Scholar logo with link to Google Scholar
OHTOMO, K. (1994) : Chapter 17. Gengo test to dai 2 gengo shutoku. (Language testing and second language acquisition). In Koike, I. (Ed.), Dai 2 gengo shutoku kekyu ni motozuku saishin no eigo kyoiku. (The latest English language education based on second language acquisition research) (pp. 300–312). Tokyo : Taishukans-hoten.Google Scholar logo with link to Google Scholar
(1996) : Komoku oto riron nyumon. (An introduction to item response theory). Tokyo : Taishukanshoten.Google Scholar logo with link to Google Scholar
PAVESI, M. (1986) : Markedness, discourse modes, and relative clause formation in a formal and an informal context. Studies in Second Language Acquisition 81, 93–105. Google Scholar logo with link to Google Scholar
QUIRK, R., GREENBAUM., LEECH, G., & SVARTVIK, J. (1985): A comprehensive grammar of the English language. London : Longman.Google Scholar logo with link to Google Scholar
SADIGHI, F. (1994) : The acquisition of English relative clauses by Chinese, Japanese, and Korean adult native speakers. International Review of Applied Linguistics 321, 141–153.Google Scholar logo with link to Google Scholar
SAJJADl, S. & TAHRIRIAN, M.H. (1992): Task variability and interlanguage use. International Review of Applied Linguistics 301, 35–44.Google Scholar logo with link to Google Scholar
SATO, C.J. (1985) : Task variation in interlanguage phonology. In S. Gass. & Madden, C.G. (Eds.). Input on Second Language Acquisition (pp.181–196). Rowley, MA : Newbury House.Google Scholar logo with link to Google Scholar
SCHACHTER, J. (1974): An error in error analysis. Language Learning 241, 205–214. Google Scholar logo with link to Google Scholar
SCHACHTER, J., TYSON, A., & DIFFEY, F.J. (1976): Learner intuitions of grammaticality. Language Learning 261, 67–76. Google Scholar logo with link to Google Scholar
SCHMIDT, M. (1980) : Coordinate structures and language universals in interlanguage. Language Learning 301, 397–416. Google Scholar logo with link to Google Scholar
SCHMIDT, R.W. (1987) : Sociolinguistic variation and language transfer in phonology. In Ioup, G. & Weinberger, S.H. (Eds.), Interlanguage phonology (pp.365–377). Rowley, MA : Newbury House.Google Scholar logo with link to Google Scholar
SHOHAMY, E. (1994) : The role of language tests in the construction and validation of second-language acquisition theories. In E.E. Tarone, Gass, S.M., and Cohen, A.D. (Eds.), Research methodology in second language acquisition (pp.133–142). Hillsdale, NJ : Lawrence Erlbaum Associates.Google Scholar logo with link to Google Scholar
SHIZUKA, T. (1993) : Task variation and accuracy predictor in interlanguage phonology production. Bulletin of Kanto-Koshin-etsu. English Language Education Society 71, 63–77.Google Scholar logo with link to Google Scholar
Statistical Product and Service Solutions Windows 7.5 version. (1966). Chicago, IL : SPSS Inc.Google Scholar logo with link to Google Scholar
STAUBLE, A. (1984) : A comparison of a Spanish-English and a Japanese English second language continuum: Negation and verb morphology. In Andersen, R. (Ed.), Second languages : A Cross-linguistic perspective (pp. 323–353). Rowley, MA: Newbury House.Google Scholar logo with link to Google Scholar
SWAIN, M. (1993) : Second language testing and second language acquisition : Is there a conflict with traditional psychometrics? Language Testing 101, 193–207. Google Scholar logo with link to Google Scholar
TAKAMIYAGI, T. (1991) : A study of task related variation in interlanguage by Japanese university students in an instruction only environment. Joetsu University of Education Kenkyu Ronshu. Bulletin of Language Studies 61, 47–64.Google Scholar logo with link to Google Scholar
TARONE, E. (1979) : Interlanguage as chameleon. Language Learning 291, 181–191. Google Scholar logo with link to Google Scholar
(1982) : Systematicity and attention in interlanguage. Language Learning 321, 142–164 Google Scholar logo with link to Google Scholar
(1983) : On the variability of interlanguage system. Applied Linguistics 41, 142–164. Google Scholar logo with link to Google Scholar
TARONE, B. (1985) : Variability in interlanguage use : A study of style-shifting in morphology and syntax. Language Learning 351, 373–403. Google Scholar logo with link to Google Scholar
TARONE, E. (1988) : Variation in interlanguage. London : Edward Arnold Publishers.Google Scholar logo with link to Google Scholar
(1989): Accounting for style-shifting in interlanguage. In S. Gass, Madden, C., Preston, D. & Selinker, L. (Eds.), Variation in second language acquisition : Psycholinguistic issues (pp. 13–21). Clevedon, England: Multilingual Matters.Google Scholar logo with link to Google Scholar
TARONE, E. & PARRISH, B. (1988) : Task-related variation in interlanguage : The case of articles. Language Learning 381, 21–44. Google Scholar logo with link to Google Scholar
TITFORD, C. (1983) : Translation and testing. International Review of Applied Linguistics 111, 312–319.Google Scholar logo with link to Google Scholar
TOMITA, Y. (1988) : Nihonjin kokosei no chukan gengo nitsuiteno ichikousatsu : Keitaiso shutoku junjo kenkyu. (A study on high school students' interlanguage : With reference to the acquisition order of morphologies). CELES Bulletin 181, 208–213.Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue