Signal detection and modulation classification are becoming increasingly important in a variety of wireless communication systems such as those involving spectrum management and electronic warfare and surveillance, among others. The majority of the signal detection and modulation classification algorithms available in the literature assume that the additive noise has a Gaussian distribution. However, while this is a good model for thermal noise, various studies have shown that the noise experienced in most radio channels, due to a variety of man-made and natural electromagnetic sources, is non-Gaussian and exhibits impulsive characteristics. Unfortunately, conventional signal processing algorithms developed for Gaussian noise conditions are known to perform poorly in the presence of non-Gaussian noise. For this reason, the main goal of this dissertation is to develop statistical signal processing algorithms for the detection and modulation classification of signals in radio channels where the additive noise is non-Gaussian.

One of the major challenges involved in the design of these algorithms is that they are expected to operate with limited or no prior knowledge of the signal of interest, the fading experienced by the signal, and the distribution of the noise added in the channel. Therefore, this dissertation develops new techniques for estimating the parameters that characterize the additive non-Gaussian noise process, as well as the fading process, in the presence of unknown signals. These novel estimators are an integral contribution of this dissertation.

The signal detection and modulation classification problems considered here are treated as hypothesis testing problems. Using a composite hypothesis testing procedure, the unknown fading and noise process parameters are first estimated and then used in a likelihood ratio test to detect the presence or identify the modulation scheme of a signal of interest. The proposed algorithms, which are developed for different non-Gaussian noise models, are shown to outperform conventional algorithms which assume Gaussian noise conditions and also algorithms based on other impulsive noise mitigation techniques.

This dissertation has three major contributions. First, in environments where the noise can be modeled using a Gaussian mixture distribution, a new expectation-maximization algorithm based technique is developed for estimating the unknown fading and noise distribution parameters. Using these estimates, a hybrid likelihood ratio test is used for modulation classification. Second, a five-stage scheme for signal detection in symmetric α stable noise environments, based on a class of robust filters called the matched myriad filters, is presented. New algorithms for estimating the noise distribution parameters are also developed. Third, a modulation classifier is proposed for environments in which the noise can be modeled as a time-correlated non-Gaussian random process. The proposed classifier involves the use of a whitening filter followed by likelihood-based classification. A new H_â filter-based technique for estimating the whitening filter coefficients is presented.