This dissertation presents a design of a robust, automatic speaker recognition (ASR) system. The ASR system is designed to work with both text-independent and text-dependent speaker recognition. Several speaker spectral features are studied to determine their contribution in term of accuracy to the system. A new algorithm is designed to label a speaker voice as either male-type voice or female-type voice. Following this division, the processing time of the speaker identification for the ASR system will be reduced by about half. Rectangular window, Hamming window, first order preemphasis filter, and many proposed spectral distances are also investigated. The principal components analysis is used to achieve high degree of female-type and male-type separation as well as the speaker recognition accuracy. Spectral features are combined to improve the recognition performance of the system. In addition, many other system components such as speech endpoint detection, automatic noise thresholds, etc. are required to build correctly in order to achieve high speaker recognition accuracy. Multi-stage decision process is used both to improve and to speed up the decision if certain criteria are met. Finally, TIMIT acoustic continuous speech corpus is used to evaluate the speaker recognition performance and the robustness of the system.