SYSTEMS AND METHODS FOR ENHANCING END-TO-END PLANNING FOR AUTONOMOUS DRIVING AND EVALUATION IN CLOSED-LOOP ENVIRONMENT

Methods and systems for training an end-to-end autonomous driving system using a vision-language planning (VLP) machine learning model in a closed-loop environment. Images associated with an environment about a vehicle are generated, and a BEV model is executed to generate a BEV view based on the images. A planning model predicts navigation trajectories based on the BEV. The VLP model enhances the system by extracting vision-based planning features, generating text prompts, and employing a language encoder to create text-based expectation features. A contrastive learning model identifies similarities between vision and text features, boosting the performance of the BEV and planning models. The system undergoes closed-loop evaluation in a simulated environment, capturing metrics to refine the autonomous driving system.