REFINEMENT TRAINING FOR MACHINE-LEARNED VEHICLE CONTROL MODEL

A machine-learned model that uses sensor and/or perception data to directly determine controls for operating an autonomous vehicle may be trained by identifying a preferred trajectory between a human-driven and vehicle-controlled trajectory, and using a first loss determined between the vehicle-controlled trajectory and the path the autonomous vehicle ultimately ended up taking in a scenario and a second loss determined between the vehicle-controlled trajectory and the human-driven trajectory to refine the machine-learned model. The machine-learned model may additionally or alternatively be refined by a learned reward model constructed by replacing one or more output heads of the machine-learned model with a regression head that is trained using performance metrics determined for the vehicle-controlled trajectory.