Automated Detection of Learning Pitfalls in Tabular Security Classifiers via Reasoning LLMs
Files
TR Number
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Security classifiers are critical for defending systems against evolving threats, yet they are frequently undermined by systemic learning flaws. Vulnerabilities such as shortcut learning and out-of-distribution (OOD) susceptibility can artificially inflate perceived performance, providing the illusion of a robust classifier while leaving real-world deployments vulnerable. While traditional diagnostic approaches, such as Trustee, exist to uncover these pitfalls, they demand intensive domain expertise and extensive manual effort to supervise the identification process. In this work, we investigate whether reasoning Large Language Models (LLMs) can overcome these bottlenecks to provide a fully automated identification pipeline. We focus on the uniquely challenging domain of tabular security classifiers, where rigid feature-level constraints and complex perturbation cost models traditionally hinder automated analysis. To this end, we propose INSPECT-R, an agentic framework that autonomously probes security classifiers and generates human-interpretable explanations for identified flaws. INSPECT-R employs a multi-agent architecture: a Researcher Agent utilizes web-augmented reasoning to automatically learn domain constraints; an Investigator Agent leverages this knowledge to isolate spurious correlations, shortcut learning, and OOD failures; and a Critique Agent validates these findings, providing iterative, reasoning-based feedback to refine the quality and validity of the Investigator's reports. Ultimately, this work represents a significant step toward minimizing the human expertise and supervision required to rigorously audit tabular classification systems.