The Waymo World Model: A New Frontier For Autonomous Driving Simulation

• Instead of training only on collected driving data like the rest of the AV industry, Waymo leverages Genie 3's pre-training on massive video datasets to simulate rare edge cases that their fleet has never encountered
• Generates both camera AND lidar outputs from world knowledge—transferring 2D video understanding into 3D point clouds unique to Waymo's sensor suite
• Three control mechanisms: driving action control for "what if" counterfactuals, scene layout control for custom scenarios, and language prompts for weather/time mutations
• Can convert any dashcam video into multimodal simulation showing how Waymo's sensors would perceive that exact scene
• Beats reconstruction methods (like 3D Gaussian Splats) which break down when simulated routes differ from recorded ones—generative approach maintains consistency on novel routes

Waymo is taking a fundamentally different approach to autonomous vehicle simulation than the rest of the industry. While most AV companies train simulation models from scratch using only their collected driving data, Waymo adapted Google DeepMind's Genie 3—a general-purpose world model pre-trained on massive, diverse video datasets—to generate driving scenarios. This transfer learning approach means they can simulate rare events like tornadoes, natural disasters, or encounters with elephants that are nearly impossible to capture at scale in reality. The key technical achievement is converting Genie 3's 2D video understanding into multimodal 3D outputs that include both camera imagery and lidar point clouds specific to Waymo's hardware suite.

The system offers three control mechanisms for scenario generation. Driving action control enables counterfactual "what if" simulations—testing whether the Waymo Driver could have driven more confidently instead of yielding in a particular situation. Scene layout control allows custom placement of road users and mutations to road layouts. Language control is the most flexible, enabling adjustments to time-of-day, weather conditions, or generation of entirely synthetic long-tail scenarios. Notably, the model can also convert regular dashcam videos into multimodal simulations, showing how Waymo's sensors would perceive that exact scene with high factuality.

The generative approach fundamentally outperforms reconstruction-based methods like 3D Gaussian Splats. When simulated routes differ significantly from originally recorded drives, reconstruction methods suffer visual breakdowns due to missing observations. The Waymo World Model maintains realism and consistency through its learned generative capabilities and world knowledge. An efficient variant enables longer rollouts with dramatic compute reduction while maintaining fidelity for large-scale simulation. This paradigm shift—from replaying observed scenarios to generating novel ones with world knowledge—creates a more rigorous safety benchmark for handling true long-tail challenges before the Waymo Driver encounters them on public roads.