Uncertainty Quantification in Security Aware Data Pipelines

Abstract

With the recent rise in connected devices through the Internet of Things and interconnected cyberphysical systems, the diversity and volume of data have expanded. Proper management of sensitive information collected and processed through data pipelines is crucial. Traditional data pipelines usually perform error analysis of the final pipeline output after a detection model. As a result, they miss malicious attacks or data corruption that occur earlier in the pipeline. Providing assurance of security throughout all stages of pipeline processing can improve credibility at a more fine-grained level. This thesis introduces a combination of data pipeline augmentation capabilities aimed at estimating the uncertainty of computations with constant monitoring of trends in shifts in data at every pipeline stage. The proposed framework integrates uncertainty quantification (UQ), data provenance tracking, sensitivity analysis, and tunable alerts to understand parameter influence on function outputs, methodically detect potential corruptions, maintain a meticulous audit trail, and prompt observers during suspicious activity. This contribution advances conventional data pipeline anomaly detection by providing combined fault-sensitive execution and full-fault traceability with continuous estimation of uncertainty for each pipeline stage.