Characterizing and Comparing the ADS Maneuver Execution Subsystem Performance of Two Vehicles

Automated driving systems (ADS) are projected to bring a plethora of benefits to society, such as enhanced road safety and heightened quality of life. However, placing one's trust in the hands of an automated system is still a large concern to society. To facilitate the large-scale adoption of ADSs, they must be stringently tested and evaluated prior to their deployment on public roadways due to their direct impact on the safety of other motorists and vulnerable road users. Currently, no standardized method of quantifying ADS performance exists, so this research project contributes to the evaluation of ADSs by developing and demonstrating a test method that solely characterizes the motion control subsystem of an ADS. The developed test method involved generating representative driving scenarios that exercised both the longitudinal and lateral control elements of an ADS. This method was then demonstrated using two test vehicles with different control system architectures by (1) defining and injecting a ground truth trajectory into the ADS, (2) characterizing the motion control subsystem by quantifying its ability to follow the ground truth path under both nominal conditions and conditions where disturbances were introduced, and (3) analyzing the response of each vehicle to characterize their respective control systems as well as identify differences between the two control architectures.

First, a set of representative driving scenarios was created to test the longitudinal and lateral control elements both in isolation and in tandem. Multiple unique design variations of each scenario were created by implementing various target speeds, accelerations, and turning radii that map to both standard and emergency maneuvers. The parameters were set to match naturalistic driving or regulatory requirements identified as part of a literature review. Next, a reference trajectory—the ground truth set of waypoints that define the position and speed of the ADS—was generated for each driving scenario. This reference trajectory was implemented using three methods: recording the waypoint trail of a human driver and creating a synthetic waypoint list mathematically or with CarMaker, a simulation platform for automobile testing (IPG Automotive 2021). Once this step was completed, the reference trajectory was inserted into the ADS to isolate the motion control system and facilitate a repeatable test input. When the test vehicle was under ADS control, the experimenter served as the designated fallback user so they could take control of the vehicle if necessary. Finally, a set of test metrics related to the operation of the ADS (lateral offset, heading error, speed error, longitudinal stop position error, and test completion percentage) were calculated using kinematic data to characterize each motion control system architecture.

The analysis of the kinematic metrics for each test scenario demonstrated that the method could effectively evaluate the performance of ADS in various scenarios and highlight the strengths and weaknesses of each system. The control system of Vehicle A consistently lagged in throttle and brake actuation and rounded corners by turning early and with a larger cornering radius. This control system also could not exceed a lateral acceleration of 3.5 m/s2 when under ADS control and limited its yaw rate to keep the lateral acceleration below this level. Consequently, this limitation caused the vehicle to turn wide for radius and speed combinations with a lateral acceleration greater than 3.5 m/s2. On the other hand, the control system of Vehicle B consistently exhibited a small delay before turning and tended to overshoot lane changes at higher lateral accelerations. Regarding disturbances, only the road grade significantly affected the response of both vehicles.