Generating synthetic driving scenes including temporary traffic control zones

Techniques are described herein for generating, using diffusion models and/or other generative models, synthetic driving scenes including traffic control zones. In various examples, a driving scene generator may provide the generative model with map data representing a driving environment, and input data representing attributes or objects of a traffic control zone (e.g., a construction zone). The diffusion model may be trained to populate the driving scene by generating agents and objects inside and outside of the traffic control zone in a realistic manner, consistent with the input data, to generate a synthetic driving scene. Synthetic driving scenes generated using generative models may be used to execute realistic simulations targeting specific driving scenarios that include construction zones and other traffic control zones.