AGENT TRAJECTORY PLANNING USING NEURAL NETWORKS

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for planning the future trajectory of an autonomous vehicle in an environment. In one aspect, a method comprises obtaining multiple types of scene data characterizing a scene in an environment that includes an autonomous vehicle and multiple agents; receiving route data specifying an intended route for the autonomous vehicle; for each data type, processing at least the data type using a respective encoder network to generate a respective encoding of the data type; processing a sequence of the encodings using an encoder network to generate a respective alternative representation for each data type; and processing the alternative representations using a decoder network to generate a trajectory planning output that comprises respective scores for candidate trajectories that represent predicted likelihoods that the candidate trajectory is closest to resulting in the autonomous vehicle successfully navigating the intended route.