A Graphical Approach to Identifying Structural Bias Using Directed Acyclic Graphs: Its Application to Two-Wave Nonequivalent Control Group Designs

Abstract

It is well known that the analysis of covariance (ANCOVA) and the change-score analysis (CSA) can produce quite different treatment-effect estimates when applied to data from two-wave nonequivalent control-group designs, a phenomenon known as the Lord's paradox. Pearl's (2009) structural causal model (SCM) provides a useful and intuitively appealing tool to address the Lord's paradox. Using the SCM, Kim and Steiner (2021) combined the backdoor criterion with the path-tracing rules and showed that it identified the exact bias for the CSA. Though they implied that this graphical causal model approach could be applied to the ANCOVA case in a similar way, they did not explicitly show the details. Therefore, in the present study, to examine their implication, I applied the graphical approach to the ANCOVA and compared the results with the bias derived by the population ordinary least squares (OLS) method (Lüdtke and Robitzsch, 2025). The comparison exhibited a discrepancy, though the core part of the bias obtained by the graphical approach was correct. Specifically, the discrepancy occurred in the terms that were proportional to the core part of the bias implied by each backdoor path. This means that, though the detection of the sources of bias and the identification of the conditions to eliminate the bias could be completed by the graphical approach, the exact quantification of the bias was not possible. To resolve this shortcoming, I applied the so-called regression anatomy formula, also known as the Frisch–Waugh–Lovell (FWL) theorem in econometrics, and found that the proportional term could be expressed as the residualization-induced scaling factor. I then extended this graphical approach to different data-generating scenarios within two-wave nonequivalent control-group designs and confirmed that it worked well in all cases. The residualization procedure makes a graphical approach self-contained to identify the exact structural bias.