The task of railroad inspection is a tedious one. It requires a lot of skilled experts and long hours of frequent on-field inspection. Automated ground equipment systems that have been developed to address this problem have the drawback of blocking the rail service during inspection process. As an alternative, using aerial imagery from a UAV, Computer Vision and Machine Learning based techniques were developed in this thesis to analyze two kinds of defects on the rail tracks. The defects targeted were missing spikes on tie plates and cracks on ties. In order to perform this inspection, the rail region was identified in the image and then the tie plate and tie regions on the track were detected. These steps were performed using morphological operations, filtering and intensity analysis. Once the tie plate was localized, the regions of interest on the plate were used to train a machine learning model to recognize missing spikes. Classification using SVM resulted in an accuracy of around 96% and varied greatly based on the tie plate illumination and ROI alignment for Lampasas and Chickasha subdivision datasets. Also, many other different classifiers were used for training and testing and an ensemble method with majority vote scheme was also explored for classification. The second category of learning model used was a multi-layered neural network. The major drawback of this method was, it required a lot of images for training. However, it performed better than feature based classifiers with availability of larger training dataset. As a second kind of defect, tie conditions were analyzed. From the localized tie region, the tie cracks were detected using thresholding and morphological operations. A machine learning classifier was developed to predict the condition of a tie based on training examples of images with extracted features. The multi-class classification accuracy obtained was around 83% and there were no misclassifications seen between two extreme classes of tie condition on the test data.