One of the major application areas for UAVs is the automated traversing and inspection of infrastructure. Much of this infrastructure is linear, such as roads, pipelines, rivers, and railroads. Rather than hard coding all of the GPS coordinates along these linear components into a flight plan for the UAV to follow, one could take advantage of computer vision and machine learning techniques to detect and travel along them. With regards to roads and railroads, two separate algorithms were developed to detect the angle and distance offset of the UAV from these linear infrastructure components to serve as control inputs for a flight controller. The road algorithm relied on applying a Gaussian SVM to segment road pixels from rural farmland using color plane and texture data. This resulted in a classification accuracy of 96.6% across a 62 image dataset collected at Kentland Farm. A trajectory can then be generated by fitting the classified road pixels to polynomial curves. These trajectories can even be used to take specific turns at intersections based on a user defined turn direction and have been proven through hardware-in-the-loop simulation to produce a mean cross track error of only one road width. The combined segmentation and trajectory algorithm was then implemented on a PC (i7-4720HQ 2.6 GHz, 16 GB RAM) at 6.25 Hz and a myRIO 1900 at 1.5 Hz proving its capability for real time UAV control. As for the railroad algorithm, template matching was first used to detect railroad patterns. Upon detection, a region of interest around the matched pattern was used to guide a custom edge detector and Hough transform to detect the straight lines on the rails. This algorithm has been shown to detect rails correctly, and thus the angle and distance offset error, on all images related to the railroad pattern template and can run at 10 Hz on the aforementioned PC.