The making of impact: Causal validity and epistemic justice in international development impact evaluation

Abstract

This dissertation investigates the manifestation and negotiation of causal validity and epistemic justice in international development impact evaluation (IDIE), with a specific focus on agricultural interventions. Drawing on Ernest House's (1980) assertion that evaluations should be not only true but also just, and Donald Campbell's (1986) reconceptualization of internal validity as "local molar causal validity," the study examines how evaluators balance methodological rigor with inclusive knowledge production. Through narrative-reflective interviews with nine evaluators representing diverse methodological perspectives, the research employs Eyal's (2013) "expertise as network" as well as research on evaluation's concepts of validity as a conceptual framework, exploring how causation and justice are part of an interconnected network of agents, concepts, devices, and institutional arrangements affect impact determination. Findings reveal multiple causality conceptions in practice, with evaluators acknowledging the counterfactual as necessary but insufficient for robust causal claims, aligning with Campbell's nuanced view of validity. Analysis utilizing Byskov's (2021) conditions for epistemic injustice and House and Howe's (2000) Deliberative Democratic Evaluation framework demonstrates that evaluators employ various strategies to promote epistemic justice while negotiating practical constraints. The study presents a four-quadrant matrix delineating elements that: (1) inhibit epistemic justice while promoting causal validity; (2) inhibit causal validity while promoting epistemic justice; (3) inhibit both; and (4) promote both. This framework illuminates potential areas of methodological and disciplinary convergence between evaluation approaches, highlighting contextual sensitivity, inclusive research practices, and methodological flexibility as practices that simultaneously advance both the causal validity championed by Campbell and the justice-oriented validity advocated by House. The findings have significant implications for evaluation practice, especially amid the structural transformations occurring within international development organizations.