SmartShield: A Proof-of-Concept Application for Augmented Reality in Driving Research

Driver behavior research has traditionally been conducted using varied modalities ranging from controlled experimental research to naturalistic observation, all of which face fundamental trade-offs among realism, control, and the maintenance of a safe environment for both drivers and other road users. New technology, particularly virtual and augmented reality (VR and AR, respectively), present opportunities for improving this balance by potentially enabling the merging of virtual, controlled stimuli with real-world driving. If such experimental tools could be developed, they have the potential to revolutionize behavioral research by combining the experimental control and safety of simulation research with the validity of on-road or test-track studies where participants drive physical vehicles in the real world.

The current effort was undertaken as a proof-of-concept challenge to implement AR glasses as a tool for driver behavior research. Researchers conducted a comprehensive review of current offerings in the AR and VR spaces and selected the candidate most likely to succeed, the Microsoft HoloLens. The HoloLens is a lightweight, head-mounted visor that combines a direct view of the world with an overlaid virtual image. As designed, the system is intended to provide support for indoor tasks in relatively static environments, such as supporting visualization in engineering and health care fields. In these controlled environments, onboard software creates three-dimensional images of persistent virtual objects that can interact with real-world surfaces using a series of cameras and positioning sensors. In theory, such technology could be extended to superimpose virtual objects onto the driving scene, the exploration of which formed the premise of this project.

Following procurement of the device, the team developed virtual objects within the Unity environment and investigated a variety of methods to overlay those objects in the real world. The process proved difficult with several challenges, including flickering and/or drifting objects, a lack of contrast in sunlight, incompatibility with externally provided GPS coordinates, and unacceptable visual range. Given that the HoloLens was intended for use in an indoor environment, relocating it to an outdoor environment revealed its limitations. While indoor performance in lower light was promising, the display was unable to compensate for the intense sunlight, and the device’s internal GPS receiver did not function at a level required for research, resulting in objects being displayed in incorrect ways. The team was able to demonstrate that the goal of using augmented reality in research is possible, but not with currently available technology. Future research should focus on utilizing newer designs with more suitability to be used outdoors in direct sunlight and at longer ranges.