Adaptive Reinforcement Learning-Based Fuzzer for 5G RRC Security Evaluation

Abstract

This thesis presents the development of an adaptive fuzzing framework leveraging the UER- ANSIM simulator to assess the security and resilience of 5G networks. The proposed approach specifically targets potential vulnerabilities in message exchange sequences, authentication procedures, and resource allocation mechanisms. It employs Q-learning to optimize fuzzing strategies across multiple protocol stack layers. The framework evaluates the impact of malicious or malformed inputs on a gNB's stability and performance by emulating multiple user devices and systematically altering key parameters in control-plane messages. A major focus of the study is the simulation of large-scale Distributed Denial-of-Service (DDoS) attacks, wherein numerous simulated UEs generate high volumes of fuzzed signaling traffic to stress the system and observe degradation. The reinforcement learning agent dynamically adjusts its attack patterns based on network feedback, tuning the injection of messages to maximize resource exhaustion and identify critical failure points. The effectiveness of the fuzzing campaign is measured through resource utilization metrics, such as CPU load and Thread count collected during experimentation. By combining adversarial testing with intelligent fuzzing techniques, the research provides important insights into the security posture of 5G infrastructure under adversarial conditions. The findings underscore the need for stronger defenses against protocol-level attacks to bolster the resilience of next-generation wireless communication systems.