SYSTEMS AND METHODS FOR ASSESSING THE PERFORMANCE OF AN AUTOMATED AUTONOMOUS DRIVING EVALUATOR

Systems and methods for assessing the performance of an automated autonomous driving evaluator are disclosed herein. One embodiment receives a driving log including at least one of raw vehicle sensor data and information derived from the raw vehicle sensor data correlated with a time index; receives ground-truth data associated with the driving log, wherein the ground-truth data includes at least one of human-initiated disengagements of an autonomous driving system and human-entered annotations indicating mistakes made by the autonomous driving system; analyzes the driving log using an automated autonomous driving evaluator to generate a report; automatically compares the report with the ground-truth data to generate a performance assessment for the automated autonomous driving evaluator; and provides the performance assessment to a development process in which the automated autonomous driving evaluator is modified based, at least in part, on the performance assessment to improve the performance of the automated autonomous driving evaluator.