Defending the Dialog: Characterizing and Mitigating Harmful Fine-Tuning Attacks in Conversational AI

Abstract

Generative AI (GenAI), especially Large Language Models (LLMs), has transformed conversational AI, enabling chatbots that converse fluently across a wide range of topics and perform diverse downstream tasks. There is growing demand to customize chatbots for specific downstream applications, and LLMs are routinely fine-tuned on application or task-specific conversational datasets. This customization paradigm, however, introduces a new attack surface. We refer to this emerging class of attacks as harmful fine-tuning attacks, in which an adversary poisons the training dataset with harmful or unsafe samples, causing the resulting chatbot to produce unsafe responses. Such attacks can cause real harm, particularly for vulnerable populations, including minorities and those with physical or mental health challenges. This dissertation systematically characterizes and defends against harmful fine-tuning attacks in conversational AI. On the attack characterization side, we focus on toxicity injection attacks, a subclass of harmful fine-tuning attacks in which an adversary injects toxic language into the training dataset. We design malicious agents that leverage GenAI to automatically inject toxic conversations into a dialog-based learning (DBL) pipeline. We then study how such attacks can poison a deployed chatbot. We further investigate the resilience of existing defenses and find that while they are effective in non-adaptive settings, they become vulnerable to adaptive adversarial attacks. Building upon these findings, on the defense side, we propose Optimus, a novel defense framework that mitigates toxicity while preserving conversational utility and seamlessly integrates into existing fine-tuning pipelines. Optimus uses safety-aligned LLMs to identify toxic samples and combines synthetic healing data with direct preference optimization (DPO) to steer the chatbot toward safe responses. Optimus remains effective even when the toxicity classifier is biased or imperfect. Optimus also remains resilient against adaptive adversarial attacks that target individual stages of the defense pipeline. Modern general-purpose chatbots, however, face a much broader range of safety harms beyond toxicity. Building upon Optimus, we propose SafetyOpus, a generalized defense framework that extends its design to a broad spectrum of safety harms spanning hate speech, drug abuse, self-harm, violence, and child abuse. SafetyOpus uses purpose-built LLM-based guardrail filters to identify harmful context-response pairs. SafetyOpus's framework restores the chatbot's safety and utility to levels comparable to those of the original base model, while also mitigating the alignment drift that arises during fine-tuning. Notably, this performance holds even when the underlying guardrail filter is imperfect, demonstrating that SafetyOpus generalizes beyond the filter's detection horizon. This dissertation takes an important step toward making chatbot customization safer by offering deployable defenses and valuable insights for future research toward developing safer and more reliable conversational AI.