Cover not available

In:Crossroads Semantics: Computation, experiment and grammar
Edited by Hilke Reckman, Lisa Lai-Shen Cheng, Maarten Hijzelendoorn and Rint Sybesma
[Not in series 210] 2017
► pp. 23–37

Get fulltext from our e-platform
JBE logo with link
PDF logo with download linkDownload Book PDF
authorization required

Chapter 2
Experimental research

Problems and opportunities in the big-data era

 | University of Amsterdam
Published online: 12 April 2017
DOI logo with link
https://doi.org/10.1075/z.210.02cre
Abstract
Experimental research in psychology, psycholinguistics or medicine provides quantitative and therefore seemingly conclusive and trustworthy evidence. However, it has been convincingly shown that most research findings are actually false. This has hardly influenced the dominant scientific evaluation system which reflects a continued trust in the unbiasedness of data by a strong reliance on simple quantifications of scientific quality and productivity, such as number of publications and number of citations. This state of affairs is remarkable in the light of a long history of strong criticism of commonly used inference methods and scientific evaluation systems, which is now backed by large-scale research projects directly questioning the reproducibility of scientific findings. This way, the large amounts of data – “big-data” – have helped to uncover some of these problematic issues, but also provided a more open attitude towards data and code sharing. In addition, novel analytic frameworks may help to better integrate empirical data with computational models.
Article outline
  • 1.Introduction
  • 2.The questionable empirical toolbox
    • 2.1Bias
    • 2.2Null hypothesis testing
    • 2.3Theory testing
  • 3.Scientific publications and evaluation
    • 3.1The economy of the publication and evaluation systems
    • 3.2Alternatives for the evaluation system
  • 4.More data, more problems?
  • Notes
  • References
References (55)
References
Bargh, John A., Mark Chen & Lara Burrows. 1996. Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology 71(2). 230–244. .Google Scholar logo with link to Google Scholar
Bornmann, Lutz, Rüdiger Mutz & Hans-Dieter Daniel. 2010. A reliability-generalization study of journal peer reviews: a multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5(12). e14331. Google Scholar logo with link to Google Scholar
Breiman, Leo. 2001. Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author). Statistical Science 16(3). 199–231. Google Scholar logo with link to Google Scholar
Button, Katherine S., John P. A. Ioannidis, Claire Mokrysz, Brian A. Nosek, Jonathan Flint, Emma S. J. Robinson & Marcus R. Munafò. 2013. Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14(5). 365–376. Google Scholar logo with link to Google Scholar
Carp, Joshua. 2012. The secret lives of experiments: Methods reporting in the fMRI literature. NeuroImage 63(1). 289–300. Google Scholar logo with link to Google Scholar
Cohen, Jacob. 1994. The earth is round (p < .05). American Psychologist 49(12). 997–1003. Google Scholar logo with link to Google Scholar
Collobert, Ronan & Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. the 25th international conference. New York, NY: ACM. Google Scholar logo with link to Google Scholar
Coyne, James C & Jacob N de Voogd. 2012. Are we witnessing the decline effect in the Type D personality literature? What can be learned? Journal of psychosomatic research 73(6). 401–407. Google Scholar logo with link to Google Scholar
Doyen, Stéphane, Olivier Klein, Cora-Lise Pichon & Axel Cleeremans. 2012. Behavioral priming: It’s all in the mind, but whose mind? PLoS ONE 7(1). e29081. Google Scholar logo with link to Google Scholar
Dreber, Anna, Thomas Pfeiffer, Johan Almenberg, Siri Isaksson, Brad Wilson, Yiling Chen, Brian A. Nosek & Magnus Johannesson. 2015. Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences 112(50). 15343–15347. Google Scholar logo with link to Google Scholar
Falk, Ruma & Charles W. Greenbaum. 1995. Significance tests die hard: The amazing persistence of a probabilistic misconception. Theory & Psychology 5(1). 75–98. Google Scholar logo with link to Google Scholar
Fanelli, Daniele. 2010. “Positive” results increase down the hierarchy of the sciences. PLoS ONE 5(4). e10068. Google Scholar logo with link to Google Scholar
Fugelsang, Jonathan A., Courtney B. Stein, Adam E. Green & Kevin N. Dunbar. 2004. Theory and data interactions of the scientific mind: Evidence from the molecular and the cognitive laboratory. Canadian Journal of Experimental Psychology/Revue canadienne de psychologie expérimentale 58(2). 86–95. Google Scholar logo with link to Google Scholar
Gigerenzer, Gerd. 2004. Mindless statistics. The Journal of Socio-Economics 33(5). 587–606. Google Scholar logo with link to Google Scholar
Gilbert, Daniel T., Gary King, Stephen Pettigrew & Timothy D. Wilson. 2016. Comment on “Estimating the reproducibility of psychological science”. Science 351(6277). 1037–1037. Google Scholar logo with link to Google Scholar
Greenberg, Steven A. 2009. How citation distortions create unfounded authority: analysis of a citation network. BMJ (Clinical research ed.) 339. b2680. Google Scholar logo with link to Google Scholar
Greenwald, Anthony G. 2012. There is nothing so theoretical as a good method. Perspectives on Psychological Science. Google Scholar logo with link to Google Scholar
Hill, Ray. 2004. Multiple sudden infant deaths – coincidence or beyond coincidence? Paediatric and perinatal epidemiology 18(5). 320–326. Blackwell Science Ltd. Google Scholar logo with link to Google Scholar
Ioannidis, John P. A. 2005. Why most published research findings are false. PLoS Medicine 2(8). e124. Google Scholar logo with link to Google Scholar
Ioannidis, John P A. 2008. Why most discovered true associations are inflated. Epidemiology 19(5). 640–648. Google Scholar logo with link to Google Scholar
Ioannidis, John P.A. 2012. Why science is not necessarily self-correcting. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 7(6). 645–654. Google Scholar logo with link to Google Scholar
Ioannidis, John P. A., Kevin W. Boyack, Henry Small, Aaron A. Sorensen & Richard Klavans. 2014. Bibliometrics: Is your most cited work your best? Nature 514(7524). 561–562. Google Scholar logo with link to Google Scholar
Kahneman, Daniel. 2013. Thinking, fast and slow. Farrar, Straus and GirouxGoogle Scholar logo with link to Google Scholar
Klein, Richard A., Kate A. Ratliff, Michelangelo Vianello, Reginald B. Adams, Štěpán Bahník, Michael J. Bernstein, Konrad Bocian, et al. 2014. Investigating variation in replicability. Social Psychology 45(3). 142–152. Google Scholar logo with link to Google Scholar
Kramer, Adam D. I., Jamie E. Guillory & Jeffrey T. Hancock. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences 111(24). 8788–8790. Google Scholar logo with link to Google Scholar
Kriegeskorte, Nikolaus. 2012. An emerging consensus for open evaluation: 18 visions for the future of scientific publishing. 1–5. Google Scholar logo with link to Google Scholar
Kriegeskorte, Nikolaus, Marieke Mur & Peter Bandettini. 2008. Representational similarity analysis – connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience 2Google Scholar logo with link to Google Scholar
Lazer, David, Ryan Kennedy, Gary King & Alessandro Vespignani. 2014. Big data. The parable of Google Flu: Traps in big data analysis. Science 343(6176). 1203–1205. Google Scholar logo with link to Google Scholar
Lindley, Dennis V. 2000. The philosophy of statistics. Journal of the Royal Statistical Society: Series D (The Statistician) 49(3). 293–337Google Scholar logo with link to Google Scholar
Loftus, Geoffrey R. 1993. A picture is worth a thousand p values: On the irrelevance of hypothesis testing in the microcomputer age. Behavior Research Methods, Instruments, & Computers 25(2). 250–256. Google Scholar logo with link to Google Scholar
Maxwell, Scott E. 2004. The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods 9(2). 147–163. Google Scholar logo with link to Google Scholar
Meehl, Paul E. 1978. Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology. Google Scholar logo with link to Google Scholar
1993. Philosophy of science: Help or hindrance? Psychological Reports 72(3). 707–733. Google Scholar logo with link to Google Scholar
Meehl, PE. 1967. Theory-testing in psychology and physics: A methodological paradox. Philosophy of science 34(2). 103–115. Google Scholar logo with link to Google Scholar
Miller, G. 2012. The smartphone psychology manifesto. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 7(3). 221–237. Google Scholar logo with link to Google Scholar
Munafo, Marcus. 2013. Deep impact: Unintended consequences of journal rank. 1–33.Google Scholar logo with link to Google Scholar
Nickerson, R. S. 2000. Null hypothesis significance testing: A review of an old and continuing controversy. Psychological methods 5(2). 241–301. Google Scholar logo with link to Google Scholar
Nuijten, Michèle B., Chris H. J. Hartgerink, Marcel A. L. M. van Assen, Sacha Epskamp & Jelte M. Wicherts. 2015. The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods. 1–22. Google Scholar logo with link to Google Scholar
Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349(6251). aac4716–aac4716. Google Scholar logo with link to Google Scholar
Pfeiffer, Thomas, Lars Bertram & John P. A. Ioannidis. 2011. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6(3). e18362. Google Scholar logo with link to Google Scholar
Rodriguez-Esteban, Raul, Ivan Iossifov & Andrey Rzhetsky. 2006. Imitating manual curation of text-mined facts in biomedicine. PLoS computational biology 2(9). e118. Google Scholar logo with link to Google Scholar
Rosenthal, Robert. 1979. The file drawer problem and tolerance for null results. Psychological Bulletin 86(3). 638–641. Google Scholar logo with link to Google Scholar
Rosenthal, Robert & John Gaito. 1963. The interpretation of levels of significance by psychological researchers. The Journal of Psychology 55(1). 33–38. Google Scholar logo with link to Google Scholar
Rutledge, Robb B., Nikolina Skandali, Peter Dayan & Raymond J. Dolan. 2014. A computational and neural model of momentary subjective well-being. Proceedings of the National Academy of Sciences 111(33). 12252–12257. Google Scholar logo with link to Google Scholar
Shmueli, Galit. 2010. To explain or to predict? Statistical Science 25(3). 289–310. Google Scholar logo with link to Google Scholar
Simmons, Joseph P., Leif D. Nelson & Uri Simonsohn. 2011. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22(11). 1359–1366. Google Scholar logo with link to Google Scholar
Simonsohn, Uri, Leif D. Nelson & Joseph P. Simmons. 2014. P-curve: A key to the file-drawer. Journal of Experimental Psychology: General 143(2). 534. Google Scholar logo with link to Google Scholar
Thompson, William C. & Edward L. Schumann. 1987. Interpretation of statistical evidence in criminal trials: The prosecutor’s fallacy and the defense attorney’s fallacy. Law and Human Behavior 11(3). Springer. 167–187. Google Scholar logo with link to Google Scholar
Tversky, Amos & Daniel Kahneman. 1971. Belief in the law of small numbers. Psychological Bulletin 76(2). 105. Google Scholar logo with link to Google Scholar
Tyler, Lorraine K, Teresa P. L. Cheung, Barry J. Devereux & Alex Clarke. 2013. Syntactic computations in the language network: Characterizing dynamic network properties using representational similarity analysis. Frontiers in Psychology 4. 271. Google Scholar logo with link to Google Scholar
Wainer, Howard. 2007. The most dangerous equation. American Scientist. Google Scholar logo with link to Google Scholar
Wicherts, Jelte M. & Marcel A. L. M. van Assen. 2012. Research fraud: Speed up reviews of misconduct. Nature 488(7413). 591–591. Google Scholar logo with link to Google Scholar
Yarkoni, Tal, Russell A. Poldrack, Thomas E. Nichols, David C. Van Essen & Tor D. Wager. 2011. Large-scale automated synthesis of human functional neuroimaging data. Nature methods 8(8). 665–670. Google Scholar logo with link to Google Scholar
Yong, Ed. 2012. Replication studies: Bad copy. Nature. Google Scholar logo with link to Google Scholar
Young, Neal S., John P. A. Ioannidis & Omar Al-Ubaydli. 2008. Why current publication practices may distort science. PLoS Medicine 5(10). e201. Google Scholar logo with link to Google Scholar
Mobile Menu Logo with link to supplementary files background Layer 1 prag Twitter_Logo_Blue