Hybrid autonomy system for autonomous and automated delivery vehicle

Provided are various systems and processes for improving last-mile delivery of real-time, on-demand orders for perishable goods. In one aspect, a method for operating an autonomous vehicle comprises receiving real-time perception data at a planner module located on the autonomous vehicle, and generating an initial motion plan based on the real-time perception data. The autonomous vehicle is maneuvered along a constrained route based on the initial motion plan without user input. An alert condition is detected from the real-time perception data, and a notification is displayed at an operator device. The notification includes a request for user input to adjust the initial motion plan. User input is received at the planner module and a modified motion plan is generated by adjusting the initial motion plan based on the user input. A virtual representation of objects identified by the AV is locally rendered for display at the operator device.