Combining latent features and non-latent features by a planning system

Techniques for generating a vehicle trajectory usable for for controlling an autonomous vehicle in an environment are described herein. For example, the techniques may include a computing device implementing a machine learned model and a statistical model in parallel in a tree search of a tree structure to predict a trajectory of the autonomous vehicle in the environment. The tree structure can process a latent representation and a non-latent representation in parallel to determine potential actions and/or trajectories for the autonomous vehicle. The tree structure of the computing device can output a trajectory for controlling the autonomous vehicle in the environment. Depending on examples, the environment is a real-world environment or a simulated environment.