Adaptive Least Mean Square (LMS) equalizers are widely used in digital communication systems primarily for their ease of implementation and lack of dependence on a priori knowledge of input signal statistics. LMS equalizers exhibit non-Wiener characteristics in the presence of a strong narrowband interference and can outperform the optimal Wiener equalizer in terms of both mean square error (MSE) and bit error rate (BER). There has been significant work in the past related to the analysis of the non-Wiener characteristics of the LMS equalizer, which includes the discovery of the shift in the mean of the LMS weights from the corresponding Wiener weights and the modeling of steady state MSE performance. BER performance is ultimately a more practically relevant metric than MSE for characterizing system performance. The present work focuses on modeling the steady state BER performance of the normalized LMS (NLMS) equalizer operating in the presence of a strong narrowband interference.

Initial observations showed that a 2 dB improvement in MSE may result in two orders of magnitude improvement in BER. However, some differences in the MSE and BER behavior of the NLMS equalizer were also seen, most notably the significant dependence (one order of magnitude variation) of the BER behavior on the interference frequency, a dependence not seen in MSE. Thus, MSE cannot be used as a predictor for the BER performance; the latter further motivates the pursuit of a separate BER model.

The primary contribution of this work is the derivation of the probability density of the output of the NLMS equalizer conditioned on a particular symbol having been transmitted, which can then be leveraged to predict its BER performance. The analysis of the NLMS equalizer, operating in a strong narrowband interference environment, resulted in a conditional probability density function in the form of a Gaussian Sum Mixture (GSM). Simulation results verify the efficacy of the GSM expression for a wide range of system parameters, such as signal-to-noise ratio (SNR), interference-to-signal (ISR) ratio, interference frequency, and step-sizes over the range of mean-square stable operation of NLMS. Additionally, a low complexity approximate version of the GSM model is also derived and can be used to give a conservative lower bound on BER performance.

A thorough analysis of the MSE and BER behavior of the Bi-scale NLMS equalizer (BNLMS), a variant of the NLMS equalizer, constitutes another important contribution of this work. Prior results indicated a 2 dB MSE improvement of BNLMS over NLMS in the presence of a strong narrowband interference. A closed form MSE model is derived for the BLMS algorithm. Additionally, BNLMS BER behavior was studied and showed the potential of two orders of magnitude improvement over NLMS. Analysis led to a BER model in the form of a GSM similar to the NLMS case but with different parameters. Simulation results verified that both models for MSE and BER provided accurate prediction of system performance for different combinations of SNR, ISR, interference frequency, and step-size.

An enhanced GSM (EGSM) model to predict the BER performance for the NLMS equalizer is also introduced, specifically to address certain cases (low ISR cases) where the original GSM expression (derived for high ISR) was less accurate. Simulation results show that the EGSM model is more accurate in the low ISR region than the GSM expression. For the situations where the derived GSM expression was accurate, the BER estimates provided by the heuristic EGSM model coincided with those computed from the GSM expression.

Finally, the two-interferer problem is introduced, where NLMS equalizer performance is studied in the presence of two narrowband interferers. Initial results show the presence of non-Wiener characteristics for the two-interferer case. Additionally, experimental results indicate that the BER performance of the NLMS equalizer operating in the presence of a single narrowband interferer may be improved by purposeful injection of a second narrowband interferer.