Challenges, Solutions, and Opportunities for Cyber Security in 5G Enabled Networks

Abstract

The swift transition toward 5G Standalone (SA) and Open Radio Access Network (O-RAN) architectures represents a paradigm shift toward highly flexible, software-defined telecommu- nications infrastructure. While these advancements offer transformative benefits in multi- vendor interoperability and network agility, they also significantly increase the threat land- scape, exposing critical vulnerabilities that have persisted across several cellular generations. This dissertation establishes a holistic security framework for 5G-enabled networks. First, a comprehensive survey of Fifth Generation (5G) SA signaling security is conducted, system- atically cataloging Access Stratum (AS) and Non-Access Stratum (NAS) messages transmit- ted in clear text during the pre-authentication window of the registration procedure. The analysis identifies critical information elements, including the 5GS Mobile Identity, User Equipment (UE) Security Capability, and Radio Resource Control (RRC) configuration pa- rameters, that remain exposed to interception and manipulation by adversaries operating Rogue Base Stations (RBSs). Second, a structured threat modeling and governance frame- work for O-RAN is proposed, informed by the MITRE FiGHT matrix and O-RAN Alliance Working Group (WG)11 security principles. Six critical threat categories are characterized, namely supply chain compromise, Fifth Generation NodeB (gNB) component compromise, rogue xApp/rApp exploitation, network sniffing and spoofing, Continuous Integration/Con- tinuous Delivery (CI/CD) pipeline exploitation, and lateral movement via container escape. A Responsible, Accountable, Supportive, Consulted, Informed (RASCI) responsibility matrix is developed to delineate security accountability across network operators, vendors, standards bodies, and testing facilities throughout the O-RAN security lifecycle. Third, a fuzz testing framework for automated 5G O-RAN vulnerability assessment is demonstrated through two complementary tools. O-FUEzzer embeds Abstract Syntax Notation One (ASN.1)-aware fuzzing directly within the OpenAirInterface5G (OAI5G) UE protocol stack to target RRC message fields, while a modified 5GReplay tool performs mutation-based fuzzing of Open Fronthaul (O-FH) interface traffic. Experimental results on an O-RAN laboratory testbed reveal that fuzzing specific O-FH fields induces complete network failure with outages exceed- ing 60 seconds, confirming the feasibility and necessity of automated, field-specific vulnera- bility testing suitable for integration into CI/CD pipelines and Open Testing and Integration Centre (OTIC) certification regimes. Fourth, a lightweight Ed25519 digital signature-based one-way authentication framework is proposed to enable UE verification of gNB identity before initiating the Random Access Procedure (RAP), directly mitigating the RBS threat. The framework embeds a 104-byte authentication payload, comprising an Ed25519 signa- ture and X.509 certificate fingerprint, within the System Information Block (SIB)1 lateNon- CriticalExtension field and incorporates timestamp-based replay protection. Experimental evaluation demonstrates 100% RBS detection across all tested scenarios, an end-to-end ver- ification latency of 3.4 ms (2.2% of the SIB1 broadcast period), full backward compatibility with legacy UE deployments, and no degradation to SIB1 broadcast periodicity. Fifth, the security analysis is extended to 5G Non-Terrestrial Networks (NTNs) by examining SIB 19, the broadcast message introduced in Third Generation Partnership Project (3GPP) Re- lease 17 that provides NTN-specific configuration and satellite ephemeris data essential for UE linkability to NTN gNBs. The threats and vulnerabilities inherent to SIB 19, includ- ing spoofing of ephemeris parameters, manipulation of NTN timing advance information, and exploitation of unprotected satellite linkability data, are systematically analyzed. The Ed25519 one-way authentication framework is then adapted to secure SIB 19, enabling UE verification of NTN gNB authenticity prior to initial access in satellite and High-Altitude Platform Station (HAPS) scenarios. Preliminary results demonstrate the viability of this extension under the unique constraints of NTN environments, including longer propagation delays and ephemeral cell visibility. Together, these contributions form a cohesive security framework that identifies vulnerabilities in 5G signaling, contextualizes them within a multi- stakeholder governance model, automates their discovery through fuzz testing, mitigates a critical class of attacks through pre-authentication base station verification, and extends these protections to non-terrestrial platforms.