Modular Predictions For Complex Human Behaviors

A device performs operations including determining a probability that a vulnerable road user (VRU) will continue on a current path (e.g., in connection with controlling an autonomous vehicle). The device receives an image depicting a vulnerable road user (VRU). The device inputs at least a portion of the image into a model, and receives, as output from the model, a plurality of probabilities describing the VRU, each of the probabilities corresponding to a probability that the VRU is in a given state. The device determines, based on at least some of the plurality of probabilities, a probability that the VRU will exhibit a behavior, and outputs the probability that the VRU will exhibit the behavior to a control system.