Examining Lord's Paradox from Causal Inference Perspective: A Simulation Study

Abstract

In two-wave nonequivalent control group designs, Change Score Analysis (CSA) and Analysis of Covariance (ANCOVA) are commonly used to estimate treatment effects. However, these analytical models can yield conflicting results in both direction and magnitude – a phenomenon known as Lord's Paradox – raising concerns regarding analytical method selection in non-experimental research. This dissertation examines Lord's Paradox from a causal inference perspective to clarify the conditions under which these discrepancies arise. Specifically, the study aims to identify the conditions under which CSA and ANCOVA yield unbiased estimates, to determine when their results converge or diverge, and to assess how measurement error in the pretest affects their performance. To address these objectives, a general data-generating model is employed that incorporates unobserved confounding variable. Analytical derivations are conducted to express the CSA and ANCOVA estimators as functions of structural parameters, enabling a precise characterization of bias. These results are complemented by simulation studies that evaluate estimator performance in several metrics. The findings indicate that CSA yields unbiased estimates only under restrictive conditions, particularly in the absence of dynamic selection or symmetric confounding (i.e., when the effects of confounding variable on the pretest and the posttest are equal). In contrast, ANCOVA achieves unbiasedness under broader conditions, although it may also produce biased estimates when its assumptions are violated. Divergence between CSA and ANCOVA results is shown to be common in the presence of dynamic selection and asymmetric confounding, and may involve not only differences in magnitude but also reversals in the direction of the estimated effect. The analysis further demonstrates that measurement error in the pretest substantially affects estimator performance, particularly for ANCOVA, as declining reliability leads to increased bias and reduced inferential accuracy.