Revisiting the Large n (Sample Size) Problem: How to Avert Spurious Significance Results

Abstract

Although large data sets are generally viewed as advantageous for their ability to provide more precise and reliable evidence, it is often overlooked that these benefits are contingent upon certain conditions being met. The primary condition is the approximate validity (statistical adequacy) of the probabilistic assumptions comprising the statistical model <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><msub><mi mathvariant="script">M</mi><mi mathvariant="bold-italic">&theta;</mi></msub><mrow><mo>(</mo><mi mathvariant="bold">x</mi><mo>)</mo></mrow></mrow></semantics></math></inline-formula> applied to the data. In the case of a statistically adequate <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mrow><msub><mi mathvariant="script">M</mi><mi mathvariant="bold-italic">&theta;</mi></msub><mrow><mo>(</mo><mi mathvariant="bold">x</mi><mo>)</mo></mrow></mrow></semantics></math></inline-formula> and a given significance level <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>&alpha;</mi></semantics></math></inline-formula>, as <i>n</i> increases, the power of a test increases, and the <i>p</i>-value decreases due to the inherent trade-off between type I and type II error probabilities in frequentist testing. This trade-off raises concerns about the reliability of declaring &lsquo;statistical significance&rsquo; based on conventional significance levels when <i>n</i> is exceptionally large. To address this issue, the author proposes that a principled approach, in the form of post-data severity (SEV) evaluation, be employed. The SEV evaluation represents a post-data error probability that converts unduly data-specific &lsquo;accept/reject <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><msub><mi>H</mi><mn>0</mn></msub></semantics></math></inline-formula> results&rsquo; into evidence either supporting or contradicting inferential claims regarding the parameters of interest. This approach offers a more nuanced and robust perspective in navigating the challenges posed by the large <i>n</i> problem.