Building Trustworthy Machine Learning Systems for Security: From Federated Learning to Agentic AI

Abstract

Federated Learning (FL) has emerged as a promising paradigm for privacy-preserving collaborative machine learning, enabling distributed devices to jointly train models without sharing raw data. This dissertation addresses critical challenges in building trustworthy and secure machine learning systems across two complementary frontiers: FL systems for network and medical security, and anomaly detection for agentic AI. Together, the five technical chapters form a progression from improving FL's utility and resilience, through applying FL to network security, to defending FL from model inversion attacks in medical settings, and finally to detecting workflow-level anomalies in modern agentic AI systems. Chapter 2 tackles data quality heterogeneity in FL to improve global model utility. Noisy-labeled and imbalanced local data among clients can severely hinder training efficiency and model convergence. We propose a quality and fairness-aware client selection mechanism based on a novel Quality-of-Model (QoM) metric that evaluates client contribution without requiring access to model updates or gradients. Our approach prioritizes clients with high-quality contributions while ensuring diversity through a sortition-inspired randomized selection process, improving convergence speed and reducing communication overhead. Chapter 3 addresses Byzantine resilience to ensure FL trustworthiness from a system security perspective. FL's distributed nature forces the central server to blindly trust local training processes, making it vulnerable to model poisoning and data poisoning attacks from malicious participants. We propose a remote attestation-based approach that regains transparency into client-side training by verifying computation integrity via Trusted Execution Environment (TEE)-generated cryptographic attestation reports, enabling the server to reject malicious updates while preserving performance under non-IID data distributions. Building on these foundations, Chapter 4 demonstrates FL applied to network intrusion detection. We propose a geometric feature learning approach that projects network traffic into a compact, well-structured representation space, combining contrastive feature learning with H-Score optimization to maximize intra-class compactness and inter-class separability. The resulting federated intrusion detection system supports both anomaly detection and precise attack type identification, including zero-day threat exploration through entropy-based uncertainty analysis. Chapter 5 addresses a distinct and critical privacy threat in FL: model inversion attacks (MIAs) that allow a malicious server to reconstruct private training data from shared model updates. We identify that the reconstruction success of all known MIAs is fundamentally bounded by the local batch size relative to the model's first-layer leakage capacity. Building on this observation, we propose Aegis, a defense that synthesizes an auxiliary dataset to push the effective batch size beyond this capacity, collapsing the server's closed-form reconstructions without modifying the FL protocol or perturbing patient data. We demonstrate the approach on medical imaging datasets. Chapter 6 extends our security focus beyond FL to the rapidly growing domain of agentic AI. Modern agentic systems execute complex tasks through long-horizon workflows involving multi-agent coordination and tool invocation, creating a new risk surface where a single injected or erroneous step propagates through downstream dependencies. We present Skynet, a workflow-level anomaly detection framework that models multi-agent execution as directed workflow graphs and learns benign behavior jointly over semantic and structural dimensions. Trained exclusively on benign workflows, Skynet detects both adversarial manipulations and intrinsic execution failures under a single decision rule, naturally extending to zero-day anomalies. Together, these five chapters provide a comprehensive framework for building trustworthy machine learning systems: improving FL utility and trustworthiness, applying FL to network and medical security, and detecting anomalies in emerging agentic AI architectures, advancing the state of the art in machine learning security in adversarial and distributed environments.