Challenging the N-Heuristic: Effect size, not sample size, predicts the replicability of psychological science.

Publisher: Public Library of Science Country of Publication: United States NLM ID: 101285081 Publication Model: eCollection Cited Medium: Internet ISSN: 1932-6203 (Electronic) Linking ISSN: 19326203 NLM ISO Abbreviation: PLoS One Subsets: MEDLINE

Original Publication: San Francisco, CA : Public Library of Science

Psychology*; Sample Size ; Humans ; Reproducibility of Results ; Heuristics

Large sample size (N) is seen as a key criterion in judging the replicability of psychological research, a phenomenon we refer to as the N-Heuristic. This heuristic has led to the incentivization of fast, online, non-behavioral studies-to the potential detriment of psychological science. While large N should in principle increase statistical power and thus the replicability of effects, in practice it may not. Large-N studies may have other attributes that undercut their power or validity. Consolidating data from all systematic, large-scale attempts at replication (N = 307 original-replication study pairs), we find that the original study's sample size did not predict its likelihood of being replicated (rs = -0.02, p = 0.741), even with study design and research area controlled. By contrast, effect size emerged as a substantial predictor (rs = 0.21, p < 0.001), which held regardless of the study's sample size. N may be a poor predictor of replicability because studies with larger N investigated smaller effects (rs = -0.49, p < 0.001). Contrary to these results, a survey of 215 professional psychologists, presenting them with a comprehensive list of methodological criteria, found sample size to be rated as the most important criterion in judging a study's replicability. Our findings strike a cautionary note with respect to the prioritization of large N in judging the replicability of psychological science.
Competing Interests: The authors have declared that no competing interests exist.
(Copyright: © 2024 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Science. 2015 Aug 28;349(6251):aac4716. (PMID: 26315443)
Proc Natl Acad Sci U S A. 2020 May 19;117(20):10762-10768. (PMID: 32366645)
PLoS One. 2019 Dec 5;14(12):e0225826. (PMID: 31805105)
PLoS Med. 2005 Aug;2(8):e124. (PMID: 16060722)
Proc Natl Acad Sci U S A. 2016 Jun 7;113(23):6454-9. (PMID: 27217556)
Nat Hum Behav. 2018 Sep;2(9):637-644. (PMID: 31346273)
Proc Natl Acad Sci U S A. 2018 Mar 13;115(11):2600-2606. (PMID: 29531091)
Am Psychol. 2015 Sep;70(6):487-98. (PMID: 26348332)
Perspect Psychol Sci. 2021 Jul;16(4):864-874. (PMID: 33412079)
J Pers Soc Psychol. 2016 Mar;110(3):431-4. (PMID: 26963765)
Psychol Sci. 2011 Nov;22(11):1359-66. (PMID: 22006061)
J Pers Soc Psychol. 2015 Jan;108(1):58-9. (PMID: 25603368)
Nat Hum Behav. 2019 Nov;3(11):1127-1128. (PMID: 31723272)
J Pers Soc Psychol. 2017 Mar;112(3):357-360. (PMID: 28221091)
Psychol Bull. 2018 Dec;144(12):1325-1346. (PMID: 30321017)
Proc Natl Acad Sci U S A. 2017 Apr 4;114(14):3714-3719. (PMID: 28320937)
Nat Hum Behav. 2017 Jan 10;1:0021. (PMID: 33954258)
PLoS One. 2014 Sep 05;9(9):e105825. (PMID: 25192357)
Biol Rev Camb Philos Soc. 2017 Nov;92(4):1941-1968. (PMID: 27879038)
Pers Soc Psychol Bull. 2019 Jun;45(6):842-850. (PMID: 30317918)
Perspect Psychol Sci. 2014 Sep;9(5):552-5. (PMID: 26186757)
PLoS One. 2014 Oct 08;9(10):e109019. (PMID: 25296159)

Date Created: 20240823 Date Completed: 20240823 Latest Revision: 20240825

20250114

PMC11343368

10.1371/journal.pone.0306911

39178270