##### Abstract

The purpose of this dissertation is to find the channel capacity in optical fiber communication systems when incoherent detection is used with single (polarization filtering) and two-polarizations (no polarization filtering).
Optical fiber systems employ photodetectors that convert optical intensity to electrical current. Bandpass vector fields may be represented by four orthogonal baseband components corresponding to two quadrature phases and two orthogonal polarizations. Intensity is proportional to the sum of the squares of these four components. In the case of a coherent receiver, a strong optical local oscillator (in phase and with same polarization as the signal) is added to the signal prior to the photodetector. This results in the removal of the quadrature phase and polarization components, and reduces to the one degree of freedom (DOF) case of signal plus local oscillator shot noise for which the Shannon channel capacity formula applies. Electrical noise following the photodetector may also be neglected if there is an optical amplifier before the photodetector in the receiver. The amplifier introduces amplified spontaneous emission noise containing both quadrature phase components and both polarizations (4 DOFs), but the 2 DOF case would result if a polarization filter were used. Although the 1 and 2 DOF cases are of less practical interest than the 4 DOF case, they provide useful benchmarks for comparing performance limits.
We evaluate both spectral efficiency limits (bps/Hz) in the limit of high and low SNR for the 1,2 and 4 DOF cases and also find the power efficiency (minimum number of photons per bit) for each of these cases. It is shown that for high SNR the spectral efficiency is the same independent of the number of DOFs and that the half-Gaussian distribution is the optimum distribution. We are able to thus obtain a compact equation for spectral efficiency which behaves in a similar way to the Shannon capacity formula but with the SNR scaled by a constant.
We also show that for low SNR the half-Gaussian distribution is not the optimum distribution as the slope of the mutual information changes with the square of SNR which would lead to the number of photons per bit becoming infinite in the limit of SNR going to zero. We use a modified half-Gaussian distribution which has a discrete component (an impulse function at the origin) and provide a simple proof that this distribution results in a mutual information that goes to zero linearly with SNR resulting in a minimum number of photons per bit. Furthermore, by increasing the magnitude of the discrete component at the origin, it is shown that the minimum number of photons per bit for the incoherent channel approaches that of the coherent channel.