3-dimensional (3D) massive multiple-input-multiple-output (MIMO)/full dimensional (FD) MIMO and application of artificial intelligence are two main driving forces for next generation wireless systems. This dissertation focuses on aspects of channel estimation and precoding for 3D massive MIMO systems and application of deep reinforcement learning (DRL) for MIMO broadcast beam synthesis. To be specific, downlink (DL) precoding and power allocation strategies are identified for a time-division-duplex (TDD) multi-cell multi-user massive FD-MIMO network. Utilizing channel reciprocity, DL channel state information (CSI) feedback is eliminated and the DL multi-user MIMO precoding is linked to the uplink (UL) direction of arrival (DoA) estimation through estimation of signal parameters via rotational invariance technique (ESPRIT). Assuming non-orthogonal/non-ideal spreading sequences of the UL pilots, the performance of the UL DoA estimation is analytically characterized and the characterized DoA estimation error is incorporated into the corresponding DL precoding and power allocation strategy. Simulation results verify the accuracy of our analytical characterization of the DoA estimation and demonstrate that the introduced multi-user MIMO precoding and power allocation strategy outperforms existing zero-forcing based massive MIMO strategies.

In 3D massive MIMO systems, especially in TDD mode, a base station (BS) relies on the uplink sounding signals from mobile stations to obtain the spatial information for downlink MIMO processing. Accordingly, multi-dimensional parameter estimation of MIMO channel becomes crucial for such systems to realize the predicted capacity gains. In this work, we also study the joint estimation of elevation and azimuth angles as well as the delay parameters for 3D massive MIMO orthogonal frequency division multiplexing (OFDM) systems under a parametric channel modeling. We introduce a matrix-based joint parameter estimation method, and analytically characterize its performance for massive MIMO OFDM systems. Results show that antenna array configuration at the BS plays a critical role in determining the underlying channel estimation performance, and the characterized MSEs match well with the simulated ones. Also, the joint parametric channel estimation outperforms the MMSEbased channel estimation in terms of the correlation between the estimated channel and the real channel.

Beamforming in MIMO systems is one of the key technologies for modern wireless communication. Creating wide common beams are essential for enhancing the coverage of cellular network and for improving the broadcast operation for control signals. However, in order to maximize the coverage, patterns for broadcast beams need to be adapted based on the users' movement over time. In this dissertation, we present a MIMO broadcast beam optimization framework using deep reinforcement learning. Our proposed solution can autonomously and dynamically adapt the MIMO broadcast beam parameters based on user' distribution in the network. Extensive simulation results show that the introduced algorithm can achieve the optimal coverage, and converge to the oracle solution for both single cell and multiple cell environment and for both periodic and Markov mobility patterns.