Scam Shield: A Hierarchical Framework for Robust and Efficient Adversarial Scam Detection

Loading...
Thumbnail Image

TR Number

Date

2026-11-05

Journal Title

Journal ISSN

Volume Title

Publisher

Virginia Tech

Abstract

Can we trust large language models (LLMs) to accurately detect scams? This thesis investigates the vulnerabilities of LLMs when facing adversarial scam messages. We created a dataset with fine-grained labels, including original scam messages, adversarial scam messages, and non-scam messages. These labels allow us to examine detection performance across different scam types in addition to the binary classification task. We evaluated GPT, Claude, and LLaMA on this dataset. Under the selected rewriting conditions, the evaluated models miss scam messages even when the deceptive intent is preserved.

To address this problem, we propose a Hierarchical Scam Detection System (HSDS). The system combines four lightweight classifiers, Decision Tree, XGBoost, Random Forest, and K-Nearest Neighbors, with a fine-tuned LLaMA 3.1 8B Instruct model. The classifiers first predict whether each message is a scam. When all four agree, the system returns their prediction. When they disagree, the message is sent to the language model. We fine-tuned this model with Low-Rank Adaptation (LoRA) using a domain-specific adversarial dataset.

The fine-tuned model achieved higher detection scores with zero-shot prompting than the evaluated few-shot baselines. The full HSDS achieved an accuracy of 0.90 and an F1-score of 0.90, with higher scores than the evaluated proprietary LLM baselines. It also reduced average inference time by approximately 56.7% compared with the fine-tuned model alone. Romance scams remained difficult to detect. Further work could examine whether semantic and contextual features improve detection of these emotionally manipulative messages.

Description

Keywords

Large Language Models, Scam Detection, Adversarial Attacks, Fine-Tuning, Multi-Model Voting

Citation

Collections