This dissertation addresses multi-layer optimization aspects of multiple input multiple output (MIMO) and deep learning-based communication systems. The initial focus is on the rate optimization for multi-user MIMO (MU-MIMO) configurations; specifically, multiple access channel (MAC) and interference channel (IC). First, the ergodic sum rates of MIMO MAC and IC configurations are determined by jointly integrating the error and overhead effects due to channel estimation (training) and feedback into the rate optimization.

Then, we investigated methods that will increase the achievable rate for parallel Gaussian IC (PGIC) which is a special case of MIMO IC where there is no interference between multiple antenna elements. We derive a generalized iterative waterfilling algorithm for power allocation that maximizes the ergodic achievable rate. We verified the sum rate improvement with our proposed scheme through extensive simulation tests.

Next, we introduce a novel physical layer scheme for single user MIMO spatial multiplexing systems based on unsupervised deep learning using an autoencoder. Both transmitter and receiver are designed as feedforward neural networks (FNN) and constellation diagrams are optimized to minimize the symbol error rate (SER) based on the channel characteristics. We first evaluate the SER in the presence of a constant Rayleigh-fading channel as a performance upper bound.

Then, we quantize the Gaussian distribution and train the autoencoder with multiple quantized channel matrices. The channel is provided as an input to both the transmitter and the receiver. The performance exceeds that of conventional communication systems both when the autoencoder is trained and tested with single and multiple channels and the performance gain is sustained after accounting for the channel estimation error.

Moreover, we evaluate the performance with increasing number of quantization points and when there is a difference between training and test channels. We show that the performance loss is minimal when training is performed with sufficiently large number of quantization points and number of channels.

Finally, we develop a distributed and decentralized MU-MIMO link selection and activation protocol that enables MU-MIMO operation in wireless networks. We verified the performance gains with the proposed protocol in terms of average network throughput.