Segmentation and Recognition of Highway Assets using Image-based 3D Point Clouds and Semantic Texton Forests

Golparvar-Fard, Mani; Balali, Vahid; de la Garza, Jesus M.

Segmentation and Recognition of Highway Assets using Image-based 3D Point Clouds and Semantic Texton Forests

dc.contributor.author	Golparvar-Fard, Mani	en
dc.contributor.author	Balali, Vahid	en
dc.contributor.author	de la Garza, Jesus M.	en
dc.contributor.department	Civil and Environmental Engineering	en
dc.date.accessioned	2013-02-18T16:58:59Z	en
dc.date.available	2013-02-18T16:58:59Z	en
dc.date.issued	2013-02-18	en
dc.description	This dataset was collected as part of research work on segmentation and recognition of highway assets in images and viedo. The research is described in detail in Journal of Computing in Civil Engineering - ASCE paper "Segmentation and Recognition of Highway Assets using Image-based 3D Point Clouds and Semantic Texton Forests". The dataset include: 12 highway asset catgegories, 3 different dataset which is divided in three groups: (a)Ground Truth images with #_#_s_GT.jpg filenames, (b)Original images with #_#_s.jpg filenames, (c)Segmented images with #_#_s_seg.jpg filenames. Folder "Test" include testing images which have not used for training. Based on results of testing images, each reconstructed 3D point cloud is categorized for one type of asset and is color coded accordingly. Note: The resolution of all images are 2573x1709 pixel. Ground Truth Images In the training images, it is assumed that for each video frame, a ground truth image is carefully generated in which the parts of the image that correspond to the asset categories are labeled and color coded accordingly. For this purpose, a comprehensive image dataset for 12 different types of asset categories is created. In this dataset, each image can contain more than one type of asset. The ground truth is labeled and color-coded for all observed types of assets and then the image is used in all appropriate corresponding training categories. Contributions The data set contains images from Virginia Tech's Smart Road. Smart Road is a unique, state-of-the-art, full scale, closed test-bed, and 2.2 mile long research facility managed by Virginia Tech Transportation Institute (VTTI) and owned and maintained by VDOT. Located at Blacksburg, VA, the Smart Road features a variety of highway assets and include specialized weather-making capabilities (rain, snow, and fog), a variable lighting test bed, pavement markings, an on-site data acquisition system, road weather information system, differential GPS system, road access and signalized intersection. The smart Road has unique capabilities and it is closed to live traffic, which made it an ideal location for data collection and experiments in this research. Disclaimer THIS DATA SET IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANT ABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The images provided above may have certain copyright issues. We take no guarantees or responsibilities, whatsoever, arising out of any copyright issue. Use at your own risk. Acknowledgements This work is supported by research grant from Institute of Critical Technologies and Applied Science (ICTAS) at Virginia Tech. With thanks to Virginia Tech Transportation Institute (VTTI) for providing access to the Smart Road.	en
dc.description.abstract	Efficient data collection of high-quantity and low-cost highway assets such as road signs, traffic signals, light poles, and guardrails is a critical element to the operation, maintenance, and preservation of transportation infrastructure systems. Despite the importance, current practice of highway asset data collection is time-consuming, subjective, and potentially unsafe. The high volume of the data that needs to be collected can also negatively impact the quality of the analysis. To address these limitations, this paper proposes a new algorithm for semantic segmentation and recognition of highway assets using video frames collected from a car-mounted camera. The proposed set of algorithms (1) takes the captured frames and using a pipeline of Structure from Motion and Multi View Stereo reconstructs a 3D point cloud model of the highway and surrounding assets; (2) using a Semantic Texton Forest classifier, each geo-registered 2D video frame at the pixel-level is segmented based on shape, texture, and color of the highway assets; and finally (3) based on the results of the 2D segmentation and a new voting scheme, each reconstructed 3D point in the cloud is also categorized for one type of asset and is color coded accordingly. The resulting augmented reality environment which integrates the color coded point clouds with the geo-registered video frames enables a user to conduct visual walk through and query different categories of assets. Experiments were performed on a challenging video dataset containing sequences filmed from a moving car on a 2.2-mile-long, two-lane highway research facility. Experimental results with an average accuracy of 76.50% and 86.75% in segmentation and pixel-level recognition of 12 types of asset categories reflect the promise of the applicability of this approach for segmentation and recognition of highway assets from image-based 3D point clouds. It also enables future algorithmic developments for 3D localization of traffic signs and other assets that are detected using the state-of-the-art vision-based methods.	en
dc.description.sponsorship	Institute of Critical Technologies and Applied Science (ICTAS) at Virginia Tech.	en
dc.identifier.doi	https://doi.org/10.1061/(ASCE)CP.1943-5487.0000283	en
dc.identifier.uri	http://hdl.handle.net/10919/19188	en
dc.relation.isreferencedby	Golparvar-Fard, M., Balali, V., and de la Garza, J. Segmentation and Recognition of Highway Assets Using Image-Based 3D Point Clouds and Semantic Texton Forests. Journal of Computing in Civil Engineering. doi: 10.1061/(ASCE)CP.1943-5487.0000283	en
dc.rights	Creative Commons CC0 1.0 Universal Public Domain Dedication	en
dc.rights.uri	http://creativecommons.org/publicdomain/zero/1.0/	en
dc.subject	High-quantity low-cost assets	en
dc.subject	Image-based 3D reconstruction	en
dc.subject	Semantic texton forest	en
dc.subject	Segmentation	en
dc.title	Segmentation and Recognition of Highway Assets using Image-based 3D Point Clouds and Semantic Texton Forests	en
dc.type	Dataset	en
dc.type.dcmitype	Dataset	en