Article published In: Methodologies for intra-sentential code-switching research
Edited by Amaia Munarriz-Ibarrola, M. Carmen Parafita Couto and Emma Vanden Wyngaerd
[Linguistic Approaches to Bilingualism 8:1] 2018
► pp. 67–97
Using two-alternative forced choice tasks and Thurstone’s law of comparative judgments for code-switching research
Published online: 18 September 2017
https://doi.org/10.1075/lab.16030.sta
https://doi.org/10.1075/lab.16030.sta
Abstract
This article argues that 2-alternative forced choice tasks and Thurstone’s law of comparative judgments (Thurstone, L. (1927). A Law of Comparative Judgment. Psychological Review, 341, 273–286. ) are well suited to investigate code-switching competence by means of acceptability judgments. We compare this method with commonly used Likert scale judgments and find that the 2-alternative forced choice task provides granular details that remain invisible in a Likert scale experiment. In order to compare and contrast both methods, we examined the syntactic phenomenon usually referred to as the Adjacency Condition (AC) (apud Stowell, T. (1981). Origins of phrase structure. PhD dissertation, Massachusetts Institute of Technology, Cambridge, MA.), which imposes a condition of adjacency between verb and object. Our interest in the AC comes from the fact that it is a subtle feature of English grammar which is absent in Spanish, and this provides an excellent springboard to create minimal code-switched pairs that allow us to formulate a clear research question that can be tested using both methods.
Article outline
- 1.Introduction
- 1.1Acceptability judgments in linguistic research
- 1.2Two-alternative forced choice judgements
- 1.3Thurstone’s law of comparative judgment
- 2.Motivation
- 2.1The AC
- 2.2Research question and hypotheses
- 2.3Methodology preview: The role of code-switching
- 3.Experiment 1: Judgments and Likert scales
- 3.1Method
- 3.1.1Participants
- 3.1.2Materials
- 3.1.3Procedure
- 3.2Results
- 3.3Preliminary Discussion
- 3.1Method
- 4.Experiment 2: 2AFC and Thurstone’s law
- 4.1First session: Method
- 4.1.1Participants
- 4.1.2Materials
- 4.1.3Procedure
- 4.2Results of session 1
- 4.2.1Unilingual sentences
- 4.2.2Code switched-sentences
- 4.2.3Calculation of the Thurstone measure
- 4.3Second session: Method
- 4.3.1Participants, Materials, and Procedure
- 4.3.2Results of session 2
- 4.1First session: Method
- 5.Discussion
- 6.Conclusions
- Notes
- Appendix
- Sentences used in Experiments 1 and 2
- Monolingual base sentences
- Block 1
- Block 2
- Monolingual base sentences
- Code switched sentences used in Experiments 1 and 2
- Condition A: Vsp+ADVsp+OBJen
- Block 1
- Block 2
- Condition B: Ven+ADVsp+OBJsp
- Block 1
- Block 2
- Condition C: Vsp+ADVen+OBJen
- Block 1
- Block 2
- Condition D: Ven+ADVen+OBJsp
- Block 2
- Condition A: Vsp+ADVsp+OBJen
- Sentences used in Experiments 1 and 2
References
References (80)
Anderson, T. (2006). Spanish-English bilinguals’ attitudes toward code-switching: proficiency, grammaticality, and familiarity. Doctoral dissertation, The Pennsylvania State University, State College, Pennsylvania.
Bader, M., & Häussler, J. (2010). Toward a model of grammaticality judgments. Journal of Linguistics, 461, 273–330.
Badiola, L., Delgado, R., Sande, A., & Stefanich, S. (2017). Code-switching attitudes and their effects on acceptability judgment tasks. Linguistic Approaches to Bilingualism, 8(1).
Bard, E. G., Robertson, D. & Sorace, A. (1996). Magnitude estimation of linguistic acceptability. Language, 721, 32–68.
Bialystok, E. (1979). Explicit and implicit judgements of L2 grammaticality. Language Learning, 291, 81–103.
Bock, R. D., & Jones, L. V. (1968). The measurement and prediction of judgment and choice. San Francisco, CA: Holden-Day.
Bošković, Ž. (1997). The Syntax of Nonfinite Complementation: An Economy Approach. Cambridge, MA: MIT Press.
Brown, T. C., & Peterson, G. L. (2009). An enquiry into the method of paired comparison: reliability, scaling, and Thurstone’s Law of Comparative Judgment. Fort Collins, CO: U.S. Department of Agriculture, Forest Service.
Brysbaert, M., New, B., & Keuleers, E. (2012). Adding Part of Speech information to the SUBTLEXUS word frequencies. Behavior Research Methods, 441, 991–997.
Bullock, B. E. & Toribio, A. J. (2012). The Cambridge Handbook of Linguistic Code-switching. Cambridge, UK: Cambridge University Press.
Caroll, J. M., Bever, T. G., & Pollack, C. R. (1981). The non-uniqueness of linguistic intuitions. Language, 571, 368–383.
Cattelan, M. (2012). Models for paired comparison data: a review with emphasis on dependent data, Statistical Science, 271, 412–433
(2000). Minimalist Inquiries: The Framework. In R. Martin, D. Michaels & J. Uriagereka (Eds.), Step by Step: Essays in Minimalist Syntax in Honor of Howard Lasnik, 89–155. Cambridge, MA: MIT Press.
Cowart, W. (1996). Experimental Syntax: Applying Objective Methods to Sentence Judgments. California: Sage Publications Inc.
David, H. A. (1988). The method of paired comparisons (2nd ed.). New York, NY: Oxford University Press.
Duchon, A., Perea, M., Sebastián-Gallés, N., Martí, A., & Carreiras, M. (2013). EsPal: One-stop shopping for Spanish word properties, Behavior Research Methods, 451, 1246–1258
Ebert, S. & Koronkiewicz, B. (2017). Monolingual stimuli as a foundation for analyzing code-switching data, Linguistic Approaches to Bilingualism, 8(1).
Edwards, L. (1957). Techniques of Attitude Scale Construction. Englewood Cliffs, New Jersey: Prentice-Hall.
Engen, T. (1971). Psychophysics, Vol. II: Scaling methods. In J. Kling & L. Riggs (Eds.), Woodworth and Schlosberg’s experimental psychology, 89–91. New York: Holt, Rinehart, & Winston.
Featherston, S. (2005a). Magnitude estimation and what it can do for your syntax: Some WH-constructions in German. Lingua, 1151, 1525–50.
(2005b). Universals and grammaticality: wh-constraints in German and English. Linguistics, 431, 667–711.
Featherson, S. (2008). Thermometer judgments as linguistic evidence. In C. M. Riehl & A. Rothe, (Eds.), Was ist linguistische Evidenz? Aachen: Shaker Verlag, 69–89.
Giancaspro, D. (2013). L2 Learners’ and Heritage Speakers’ Judgments of Code-Switching at the Auxiliary-VP Boundary. Selected Proceedings of the 16th Hispanic Linguistics Symposium, J. Cabrelli Amaro et al. (Eds.), 56–69. Somerville, MA: Cascadilla Proceedings Project.
Gibson, E., Piantadosi, S., & Fedorenko, K. (2011). Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments. Language and Linguistics Compass 51, 509–524.
Gigerenzer, G., & Richter, H. (1990). Context effects and their interaction with development: Area judgments. Cognitive Development, 51, 235–264.
Gigerenzer, G., Krauss, S. & Vitouch, O. (2004). The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences. Thousand Oaks, CA: Sage.
Jones, L. V., & Thissen, D. A. (2007). A History and Overview of Psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics, volume 26: Psychometrics, 1–27. New York, NY: Elsevier.
Kratzer, A. (1996). Severing the External Argument from its Verb, in J. Rooryck & L. Zaring (Eds.), Phrase Structure and the Lexicon. Dordrecht: Kluwer Academic Publishers.
López, L. (2012). Indefinite objects: scrambling, choice functions and differential marking. Cambridge, Mass: MIT Press.
Luce, R. D. (2002). A psychophysical theory of intensity proportions, joint presentations, and matches. Psychological Review, 1091. 520–532.
MacSwan, J. (1999). A minimalist approach to intrasentential code switching: Spanish-Nahuatl bilingualism in Central Mexico. New York: Garland.
Montag, E. D. (2006). Empirical formula for creating error bars for the method of paired comparisons. Journal of Electronic Imaging, 151, 222–230.
Mohan, B. A. (1977). Acceptability testing and fuzzy grammar. In S. Greenbaum (Ed.), Acceptability in language, 133–148. The Hague: Mouton
Muysken, P. (2000). Bilingual Speech: A typology of code-mixing. Cambridge: Cambridge University Press
Munarriz, A. & Parafita Couto, M. C. (2014). ¿Cómo estudiar el cambio de código ? Incorporación de diferentes metodologías en el caso de varias comunidades bilingües. Lapurdum, 181, 43–73
Nagata, H. (1988). The relativity of linguistic intuition: The effect of repetition on grammaticality judgments. Journal of Psycholinguistic Research, 1711, 1–17.
Narens, L. (1996). A theory of ratio magnitude estimation. Journal of Mathematical Psychology, 401, 109–129
Onar Valk, P. (2014). Convergent developments in Dutch Tukish word order – A comparative study using ‘elicited production’ and ‘judgment’ data: Converging evidence?, Applied Linguistics Review, 51, 353–374.
Oxford University Language Centre. “Placement Tests.” lang.ox.ac.uk. [URL] (accessed July 1st, 2015)
Parafita Couto, M. C., Deuchar, M., & Fusser, M. (2015). How do Welsh-English bilinguals deal with conflict? Adjective-noun order resolution. In: G. Stell, K. Yakpo (Eds.), Code-switching at the crossroads between structural and sociolinguistic perspectives, 65–84. Boston: Mouton de Gruyter.
Parafita Couto, M. C., Munarriz, A., Epelde, I., Deuchar, M., & Oyharçabal, B. (2015). Gender conflict resolution in Spanish-Basque mixed DPs. Bilingualism, Language and Cognition, 181, 304–323.
Parraga, C. A. (2015). Perceptual Psychophysics. In G. Cristobal, M. Keil, & L. Perrinet (Eds.), Biologically-Inspired Computer Vision: Fundamentals and Applications, 81–108. New York, NY: Wiley.
Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 461, 1023–1031
Pollock, J. Y. (1989). Verb Movement, Universal Grammar, and the Structure of IP. Linguistic Inquiry, 201, 365–424
Ross, J. R. (1979). Where’s English? In C. J. Fillmore, D. Kemper, & W. S. Wang (Eds.), Individual differences in language ability and language behavior, 127–163. New York: Academic Press.
Serratrice, L., Sorace, A., Filiaci, F., & Baldo, M. (2009). Bilingual children’s sensitivity to specificity and genericity: Evidence from metalinguistic awareness. Bilingualism: Language and Cognition, 121, 239–257.
Schütze, C. T. (2016). The empirical base of linguistics. Grammaticality judgments and linguistic methodology. Chicago & London: University of Chicago Press.
Schütze, C. T., & Sprouse, J. (2013). Judgment Data. In R. J. Podesva and D. Sharma (Eds.), Research Methods in Linguistics, pp. 27–50. Cambridge, England: Cambridge University Press.
Shin, F., Goodall, G., Michel, D. & Beecher, H. (2012). Is Magnitude Estimation worth the trouble? In J. Choi, E. Alan Hogue, J. Punske, D. Tat, J. Schertz & A. Truman (Eds.), Proceedings of the 29th West Coast Conference on Formal Linguistics (WCCFL29). 328–336. Somerville, MA: Cascadilla Proceedings Project.
Snow, C., & Meijer, G. (1977). On the secondary nature of syntactic intuitions. In S. Greenbaum (Ed.), Acceptability in language, 163–177. The Hague, the Netherlands: Mouton.
Sorace, A. (1996). The use of acceptability judgments in second language acquisition research. In W. C. Ritchie and T. K. Bhatia (Eds.), Handbook of second language acquisition, pp. 375–409. San Diego, CA: Academic Press.
(2010). Magnitude estimation in language acquisition research. In S. Unsworth & E. Blom (Eds.), Experimental Methods in Language Acquisition, pp. 57–72. Amsterdam: John Benjamins.
Sorace, A., Serratrice, L., Filiaci, F., & Baldo, M. (2009). Discourse conditions on subject pronoun realization: Testing the linguistic intuitions of older bilingual children. Lingua, 1191, 460–477.
Sprouse, J. (2007). Continuous acceptability, categorical grammaticality, and experimental syntax. Biolinguistics, 11, 118–129.
(2011). A Test of the Cognitive Assumptions of Magnitude Estimation: Commutativity does not Hold for Acceptability Judgments. Language, 871, 274–288.
Sprouse, J. & Almeida, D. (2011). Power in acceptability judgment experiments and the reliability of data in syntax. Ms., University of California, Irvine & Michigan State University.
(2012). The empirical status of data in syntax: A reply to Gibson and Fedorenko. Language and Cognitive Processes. iFirst. 1–7.
Sprouse, J., Schütze, C. T. & Almeida, D. (2013). A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010. Lingua, 1341, 219–248.
(1956). The direct estimation of sensory magnitudes: loudness. The American journal of psychology, 691, 1–25
Stowell, T. (1981). Origins of phrase structure. PhD dissertation, Massachusetts Institute of Technology, Cambridge, MA.
Stokes, W. (1974). All of the work on quantifier-negation isn’t convincing. In M. W. La Galy, R. A. Fox, & A. Bruck (Eds.), Papers from the tenth regional meeting, Chicago Linguistic Society, 692–700. Chicago: Chicago Linguistic Society.
Tikofsky, R. S., & Reiff, D. G. (1970). Structural apperception in the absence of syntactic constraints, Language and Speech, 131, 240–253.
Tsukida, K. & Gupta, M. R. (2011). How to Analyze Paired Comparison Data (UWEE Technical Report Number UWEETR-2011–0004) Seattle, University of Washington.
Toribio, A. J. (2001). On the emergence of bilingual code-switching competence. Bilingualism: Language and Cognition, 41, 203–231.
Cited by (37)
Cited by 37 other publications
Delgado, Rodrigo & Bryan Koronkiewicz
Gernsbacher, Morton Ann, Christen E. Seyl & Amanda M. Cox
Jacobs, Oliver, Farid Pazhoohi, Grayson Mullen & Alan Kingstone
Jiménez-Fernández, Ángel L.
Olson, Daniel J.
Thwaites, Peter, Pauline Jadoulle & Magali Paquot
Wu, Ruixue, M. Carmen Parafita Couto & Niels O. Schiller
Alves, Matheus Gomes & Adriana Leitão Martins
Feleke, Tekabe Legesse
MacSwan, Jeff & Kellie Rolstad
Almeida, Andre, Weicong Li, Emery Schubert, John Smith & Joe Wolfe
Balam, Osmer, Hans Stadthagen-González, Eva Rodríguez-González & María del Carmen Parafita Couto
Hoot, Bradley & Tania Leal
Bellamy, Kate & M. Carmen Parafita Couto
2022. Gender assignment in mixed noun phrases. In The Acquisition of Gender [Studies in Bilingualism, 63], ► pp. 13 ff.
Bellamy, Kate, Hans Stadthagen-Gonzalez & M. Carmen Parafita Couto
Cocchi, Gloria & Cristina Pierantozzi
De Nicolás, Irati & Luis López
Greidanus Romaneli, Miriam, Ivo H. G. Boers & M. Carmen Parafita Couto
Balam, Osmer, María del Carmen Parafita Couto & Hans Stadthagen-González
Gerasimova, Anastasia & Ekaterina Lyutikova
Kootstra, Gerrit Jan, Ton Dijkstra & Janet G. van Hell
Suurmeijer, Luuk, M. Carmen Parafita Couto & Marianne Gullberg
Vaughan-Evans, Awel, Maria Carmen Parafita Couto, Bastien Boutonnet, Noriko Hoshino, Peredur Webb-Davies, Margaret Deuchar & Guillaume Thierry
Koronkiewicz, Bryan
Pablos, Leticia, M. Carmen Parafita Couto, Bastien Boutonnet, Amy de Jong, Marlou Perquin, Annelies de Haan & Niels O. Schiller
2019. Adjective-noun order in Papiamento-Dutch code-switching. Linguistic Approaches to Bilingualism 9:4-5 ► pp. 710 ff.
Pablos, Leticia, M. Carmen Parafita Couto, Bastien Boutonnet, Amy de Jong, Marlou Perquin, Annelies de Haan & Niels O. Schiller
2021. Adjective-noun order in Papiamento-Dutch code-switching. In Psycholinguistic Approaches to Production and Comprehension in Bilingual Adults and Children [Benjamins Current Topics, 117], ► pp. 211 ff.
Parafita Couto, Maria Carmen & Hans Stadthagen-Gonzalez
Saad, George, Marian Klamer & Francesca Moro
Sheehan, Michelle, Martin Schäfer & Maria Carmen Parafita Couto
Stadthagen-González, Hans, M Carmen Parafita Couto, C Alejandro Párraga & Markus F Damian
Bellamy, Kate, M. Parafita Couto & Hans Stadthagen-Gonzalez
Camacho, José
Ebert, Shane & Bryan Koronkiewicz
2018. Monolingual stimuli as a foundation for analyzing code-switching data. Linguistic Approaches to Bilingualism 8:1 ► pp. 25 ff.
Munarriz-Ibarrola, Amaia, Maria del Carmen Parafita Couto & Emma Vanden Wyngaerd
2018. Methodologies for intra-sentential code-switching research. Linguistic Approaches to Bilingualism 8:1 ► pp. 1 ff.
Valdés Kroff, Jorge R., Rosa E. Guzzardo Tamargo & Paola E. Dussias
2018. Experimental contributions of eye-tracking to the understanding of comprehension processes while hearing and reading code-switches. Linguistic Approaches to Bilingualism 8:1 ► pp. 98 ff.
This list is based on CrossRef data as of 24 november 2025. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
