Contributions to Modeling and Analysis of Complex Data for Engineering and AI Systems

Abstract

Engineering and AI systems increasingly operate in settings characterized by high dimensionality, structural complexity, and practical constraints, where statistical methods must support consequential decisions in domains such as power systems and global health. This dissertation develops both methodological and applied contributions under a unifying theme: the modeling and analysis of complex data for engineering and AI systems. The first methodological contribution addresses variable selection in generalized linear models with conditional main effects (CMEs), which provide an interpretable alternative to generic interaction terms by representing the effect of one factor at specific levels of another. To support structured effect selection in this setting, I develop an adaptive bi-level framework that extends the cmenet family of penalties to generalized linear models. The proposed method uses adaptive weights to regulate the coupling among related CMEs, encouraging coherent selection of grouped conditional effects while still allowing individual terms to enter or leave the model when the signal is localized. An efficient coordinate descent algorithm is developed for estimation, and simulation studies show improved performance relative to standard penalized regression approaches under complex conditional effect structures. A gene association case study further demonstrates the interpretability and predictive value of the proposed method. The second methodological contribution concerns experimental design under treatment cardinality constraints, in which each run is restricted to contain only a pre-specified number of active factors. Such constraints arise naturally in engineering and screening experiments but are rarely incorporated explicitly into the literature of design construction. This dissertation develops an optimal sparse projection design framework for binary experiments with fixed run-wise cardinality, introducing criteria that characterize projection quality and aliasing under the constraint and using them to guide both algebraic and algorithmic design construction. Simulation studies and engineering-motivated examples show that the resulting designs achieve substantially improved low-dimensional projection performance compared with naive adaptations of traditional designs. The applied chapters show how statistical methods can support interdisciplinary decision-making. In global health, I conduct a model-based cost and cost-effectiveness analysis of ivermectin mass drug administration for malaria control in Kwale County, Kenya, using data from a Phase III cluster-randomized trial. A decision-analytic framework links trial outcomes to provider and household costs and to disability-adjusted life years, while deterministic and probabilistic sensitivity analyses are used to quantify uncertainty. The results provide evidence on the potential value of ivermectin-based malaria interventions from health system, household, and societal perspectives. In power systems, I study machine-learning-based electricity load forecasting in the PJM Interconnection under rapid demand growth associated with data centers, identify mechanisms of forecast bias and error propagation, and develop a two-stage correction framework that substantially improves multi-horizon forecasting performance at both system and zonal levels. Overall, these contributions advance statistical methodology for structured modeling and constrained design while illustrating the broader role of statistics in producing interpretable, robust, and decision-relevant analyses for complex engineering and public health systems.