Is N-Hacking Ever OK? The consequences of collecting more data in pursuit of statistical significance.

Publisher: Public Library of Science Country of Publication: United States NLM ID: 101183755 Publication Model: eCollection Cited Medium: Internet ISSN: 1545-7885 (Electronic) Linking ISSN: 15449173 NLM ISO Abbreviation: PLoS Biol Subsets: MEDLINE

Original Publication: San Francisco, CA : Public Library of Science, [2003]-

Research Design*/standards ; Data Accuracy*; Reproducibility of Results

Upon completion of an experiment, if a trend is observed that is "not quite significant," it can be tempting to collect more data in an effort to achieve statistical significance. Such sample augmentation or "N-hacking" is condemned because it can lead to an excess of false positives, which can reduce the reproducibility of results. However, the scenarios used to prove this rule tend to be unrealistic, assuming the addition of unlimited extra samples to achieve statistical significance, or doing so when results are not even close to significant; an unlikely situation for most experiments involving patient samples, cultured cells, or live animals. If we were to examine some more realistic scenarios, could there be any situations where N-hacking might be an acceptable practice? This Essay aims to address this question, using simulations to demonstrate how N-hacking causes false positives and to investigate whether this increase is still relevant when using parameters based on real-life experimental settings.
Competing Interests: The author has declared that no competing interests exist.
(Copyright: © 2023 Pamela Reinagel. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Arch Soc Esp Oftalmol. 2017 Dec;92(12):e75-e77. (PMID: 28890235)
Infant Behav Dev. 2019 Feb;54:166-176. (PMID: 30470414)
PLoS Biol. 2015 Mar 13;13(3):e1002106. (PMID: 25768323)
Psychon Bull Rev. 2014 Apr;21(2):268-82. (PMID: 24002963)
Front Psychol. 2016 Sep 22;7:1444. (PMID: 27713723)
Account Res. 2023 Mar 3;:1-5. (PMID: 36866641)
Philos Trans A Math Phys Eng Sci. 2009 Nov 13;367(1906):4361-83. (PMID: 19805449)
Osteoarthritis Cartilage. 2018 Nov;26(11):1409-1411. (PMID: 30096356)
Perspect Psychol Sci. 2009 May;4(3):274-90. (PMID: 26158964)
Bioessays. 2020 Jan;42(1):e1900189. (PMID: 31755115)
PLoS One. 2019 May 15;14(5):e0216125. (PMID: 31091251)
Elife. 2020 Sep 15;9:. (PMID: 32930092)
J Clin Psychiatry. 2021 Feb 18;82(1):. (PMID: 33999541)
PLoS Med. 2005 Aug;2(8):e124. (PMID: 16060722)
Res Integr Peer Rev. 2019 Nov 29;4:24. (PMID: 31798975)
Sci Eng Ethics. 2020 Jun;26(3):1531-1547. (PMID: 31981051)
PLoS Biol. 2018 Apr 4;16(4):e2005282. (PMID: 29617358)
Ann Intern Med. 1999 Jun 15;130(12):1005-13. (PMID: 10383350)
Perspect Psychol Sci. 2014 Nov;9(6):641-51. (PMID: 26186114)
J Empir Res Hum Res Ethics. 2019 Jul;14(3):209-218. (PMID: 30943835)
Sci Eng Ethics. 2021 Jun 29;27(4):41. (PMID: 34189653)
Perspect Psychol Sci. 2012 Nov;7(6):632-8. (PMID: 26168122)
Signif (Oxf). 2020 Jul 29;17(4):8-9. (PMID: 37250179)
Epidemiology. 2001 May;12(3):295-7. (PMID: 11337600)
Bioessays. 2018 Jul;40(7):e1800055. (PMID: 29742282)
Naunyn Schmiedebergs Arch Pharmacol. 2014 Nov;387(11):1017-23. (PMID: 25213136)
Nat Commun. 2020 Jan 17;11(1):358. (PMID: 31953411)
Psychol Sci. 2011 Nov;22(11):1359-66. (PMID: 22006061)
Science. 2021 Jul 16;373(6552):265. (PMID: 34437132)
R Soc Open Sci. 2014 Nov 19;1(3):140216. (PMID: 26064558)
Front Psychol. 2022 Dec 13;13:1020770. (PMID: 36582318)
Nat Commun. 2019 Apr 23;10(1):1921. (PMID: 31015469)
Perspect Psychol Sci. 2021 Jul;16(4):717-724. (PMID: 33593151)
PLoS One. 2018 Jul 16;13(7):e0200303. (PMID: 30011289)

Date Created: 20231101 Date Completed: 20231121 Latest Revision: 20231121

20240628

PMC10619921

10.1371/journal.pbio.3002345

37910647