AI Dash Cam Performance Testing: Final Report

Abstract

The objective of this project was to assess the Nauto® driver monitoring system (DMS) technology for alert performance. The study tested the technology’s alert performance for six risky driving behaviors as performed in a heavy truck and five driver maneuvers performed during high-risk driving scenarios in a light-duty vehicle. The task types were selected by the sponsor. VTTI evaluated DMS performance by calculating rates of in-cabin audible alerts, rates of alerts recorded on the virtual platform’s dashboard, and time to alert (TTA) from the start of the tested driving behavior or from time to collision, depending on the test scenario. The study included a test-track experiment to determine the rate at which the system provided in-cab audible alerts in response to specific driving behaviors and maneuvers. The test-track experiment driving behaviors were performed between September 20, 2024, and December 12, 2024. The first six specific driving behaviors were performed in a Class 8 tractor (i.e., close following of a lead vehicle; making an outgoing phone call on a handheld smartphone; sending an outgoing text message; discreetly using a smartphone in lap; not wearing a seatbelt; and performing a rolling stop through a stop sign). The remaining driving maneuvers were performed in a light-duty commercial vehicle (i.e., approaching a stopped vehicle target to elicit an external scene-only forward collision warning [FCW]; approaching a vulnerable road user [VRU] target to elicit an external scene-only FCW; approaching a stopped vehicle target while texting to elicit a FCW based on the fusion of an external FCW paired with driver distraction; approaching a VRU target while texting to elicit an FCW based on the fusion of an external FCW paired with driver distraction; close following of a lead vehicle while sending an outgoing text message). The Nauto DMS was tested in a single installation position in each study vehicle. Testing took place on VTTI’s Smart Roads test track. The performed behaviors and maneuvers were tested under daytime and nighttime lighting conditions. The stopped vehicle was an inflatable target from a tow-behind vehicle model system. The VRU targets included an adult pedestrian target and a motorcycle target, both models meeting Euro New Car Assessment Program standards. During testing, the stationary pedestrian target’s arms and legs moved in a motion consistent with walking across the roadway; a remote control was used to operate the model. In-cab alerts, dashboard alerts, and TTA were recorded for every trial. The following results provide a summary of key findings and do not provide a comprehensive review of all analyses. Further details can be found in the Results section of the report. In the Class 8 tractor, the system provided audible alerts for all trials of tested driver-distraction-related tasks (i.e., making an outgoing phone call on a handheld smartphone, sending an outgoing text message, and discreetly using a handheld smartphone in lap) and the seat belt use task. For the rolling stop task, in-cabin alerts were provided in 95% of all trials. The audible alert rate for the close following of a lead vehicle task was 50% over all trials. In the light-duty commercial vehicle, the system provided audible alerts in at least 80% of daytime trials for external scene fusion and external scene-only FCW tasks, regardless of target. Nighttime trials of these tasks had audible alert rates of 50% or below, with no alerts provided for external scene-only FCW using a pedestrian. For FCW tasks, 75% to 93% of trials with alerts had alerts provided prior to the swerve point, depending on the task. Close following of a lead vehicle while texting had an overall alert rate of 95%. Average TTA for the driver distraction-related tasks performed in the Class 8 tractor was under 5 seconds, regardless of lighting condition. Average TTA for the rolling stop task was 3.36 seconds. Average TTA for the seat belt use task was 12.95 seconds. For the close following of a lead vehicle task, the range in TTA across trials was 16 seconds for daytime trials and over 20 seconds for nighttime trials. For FCW tasks in the light-duty commercial vehicle, TTA was calculated as time between the start of the alert and the vehicle reaching the target. For the stopped vehicle target tasks, the overall average TTA was 1.99 seconds for external scene-only FCW and 2.47 seconds with driver distraction. For the pedestrian target tasks, overall average TTA was 2.67 seconds without distraction and 2.30 seconds with driver distraction. For the motorcycle target tasks, the overall average TTA was 3.93 seconds without distraction and 4.24 seconds with driver distraction. Close following of a lead vehicle with distraction had an average TTA of 6.27 seconds in daytime trials and 9.06 seconds in nighttime trials. For all trials performed in the Class 8 tractor, the dashboard alert rate matched that of the in-cab alert rate. In the commercial light vehicle, a trial of external scene FCW using a motorcycle target that did not have an in-cabin audible alert was included on the dashboard with the appropriate alert type. For all other tasks and trials performed in the light-duty commercial vehicle, the dashboard alert rate matched that of the in-cab alert rate. The current study used a validated approach, with repeatable methods, and was performed in a controlled setting to collect data on the performance of the Nauto DMS. However, it is also important to document study limitations, which inform the accurate interpretation of the presented results and the ability of the results to be extrapolated outside of the tested conditions. VTTI does not endorse any dash cam used or not used in this study. The results obtained are based on controlled testing in specific conditions using only one driver. The current study assessed the system’s performance for three specific metrics and was not a holistic assessment of system performance, effectiveness of the DMS on changing driver behavior or crash avoidance and prevention, or carrier/driver acceptance of the device.