Xu, Jiarui2025-05-312025-05-312025-05-30vt_gsexam:43366https://hdl.handle.net/10919/134944Next-generation (NextG) cellular networks are envisioned to integrate artificial intelligence (AI) and machine learning (ML) into the air interface to meet increasingly stringent performance demands. Multiple-input multiple-output (MIMO) and its variants, such as massive MIMO, have been key enablers across successive generations of cellular networks with evolving design challenges and complexities. However, developing AI/ML-based solutions for MIMO operations in NextG systems is challenging due to the large number of possible system configurations, dynamic channel environments, and real-time operation adaptations. The highly dynamic nature of wireless environments and the operation adaptations necessitate AI/ML solutions to adapt to rapid channel variations and operation changes on a sub-millisecond basis. To this end, this dissertation develops various online and real-time AI/ML-based methods for the receive processing task in the NextG air interface with a focus on the MIMO orthogonal frequency-division multiplexing (OFDM) symbol detection, orthogonal time frequency space (OTFS) symbol detection, and MIMO-OFDM channel estimation task. To enable efficient learning, domain knowledge, such as the symmetric structure of the modulation constellation, the delay-Doppler (DD) domain input-output relationship, and the channel statistics, is inherently embedded in the design of the neural network. All introduced algorithms achieve outstanding performance while learning from only a limited number of over-the-air (OTA) training pilots on a 5G slot basis. This dissertation highlights the critical role of integrating AI/ML with domain knowledge in cellular communication systems, paving the way for the deployment of AI/ML-based techniques in NextG networks.ETDenIn CopyrightMachine LearningOnline LearningSymbol DetectionChannel EstimationReceive ProcessingMIMO-OFDMOTFSChannel EqualizationNeural NetworkDissertation