ML Engineer (3D/4D World Modeling, Simulation)

Waymo LLC; Mountain View (CA); Hybrid; USD 240,000 - 360,000; Full time

Benefits offered by this job

Medical, dental, and vision insurance

Annual bonus and equity opportunities

Hybrid work model with remote options

Job summary

Waymo LLC is seeking a senior leader to define and drive the architectural roadmap for large-scale 4D world models and multimodal sensor simulation. You will bridge multimodal pre-training across 4D video, LiDAR, and Radar, and scale training across TPU/GPU clusters to enable safe, high-fidelity driving world simulations.

The role reports to a Senior Staff Engineering Manager and requires mentoring senior engineers, establishing robust training recipes, and ensuring production-ready simulation

Qualifications

• PhD or MS in Computer Science, Machine Learning, Robotics, Computer Vision, or a related quantitative field.
• Expert-level proficiency in Python and modern distributed ML frameworks.
• 8+ years of industry or research leadership in ML architectures.

Responsibilities

• Define and drive the technical roadmap for foundational 4D world modeling and related architectures.
• Train and scale foundation video and physical world models on multi-node clusters (TPU/GPU).
• Design and implement post-training and alignment methods (SFT, RLAIF, distillation).
• Collaborate with onboard perception, planner, and AI foundation teams for deployment-ready simulations.

Skills

Python

Distributed ML

C++

Framesharing frameworks?

Education

PhD or MS in Computer Science / ML / Robotics / Computer Vision

Tools

JAX/Flax

XLA/Pallas

PyTorch

Megatron-LM

Pathways

DeepSpeed

Job description

• The Simulator Team at Waymo builds state-of-the-art simulations of realistic environments for testing, training, and validation of the Waymo Driver
• Our team is a multidisciplinary group of machine learning (ML) engineers, software engineers, and research scientists pushing the boundaries of autonomous systems
• We develop industry-leading simulation platforms using large-scale generative and physical world modeling algorithms to model the physical world—encompassing dynamic agents, complex road topologies, multi-modal traffic systems, weather phenomena, and the vehicle’s full sensor suite (Camera, Lidar, Radar)
• To pioneer the next frontier of fidelity, scalability, and physical controllability, we are expanding our focus on foundation world models
• In this role, you will define and drive the technical roadmap for large-scale generative architectures, bridging cutting-edge multimodal pre-training (across 4D video, LiDAR, and Radar), spatiotemporal tokenization, and causal post-training/distillation to generate interactive, multi-platform, safety-critical driving worlds for closed-loop evaluation and onboard model training
• In this role, you will report to a Senior Staff Engineering Manager
• Own the Architectural Roadmap: Drive the technical vision and long-term roadmap for foundational 4D world modeling, unified spatiotemporal tokenizers (3D/4D VAEs), video generation architectures, and generative sensor simulation across Waymo — enabling zero-shot generalization across next-generation vehicle platforms, sensor rigs, and international geographies
• Lead Large-Scale Pre-Training: Architect, train, and scale foundation video and physical world models (including Sparse MoE and long-context block-sparse attention architectures) across distributed TPU/GPU clusters, establishing efficient training recipes, multimodal data curation strategies, and scaling laws
• Drive Advanced Post-Training & Alignment: Design and implement robust post-training paradigms, including Supervised Fine-Tuning (SFT), Reinforcement Learning with AI/Critic Feedback (RLAIF), model merging/souping, and 3D/semantic conditioning to guarantee strict physical realism, counterfactual pose-divergence stability, and fine-grained control over safety-critical scene elements (signage, traffic signals, VRU gestures, and adverse weather)
• Integrate Multimodal & Physical Signals: Develop action-conditioned, 3D-grounded physical world models and multi-view generative pipelines that couple explicit geometric and physical priors (4D scene graphs, occupancy, vehicle kinematics, and interactive agent dynamics) with neural synthesis across Camera, LiDAR, and Radar—faithfully modeling physical sensor transport, occlusions, and causal state transitions
• Optimize for Production & Serving: Scale inference and evaluation frameworks into production simulation infrastructure, applying few-step and causal autoregressive distillation (e.g., Distribution Matching Distillation [DMD], Self-Forcing, Consistency/Flow distillation), quantization, and KV-cached streaming inference for 10 Hz high-throughput closed-loop validation and RL gyms
• Foster Research-to-Production Impact: Partner closely with Onboard Perception, Planner, and AI Foundation labs across Waymo and Alphabet (e , Google DeepMind) to adapt frontier world models into deployment-ready simulation environments that achieve downstream perception/policy parity for rare-event reproduction, synthetic data training, and closed-loop RL
• Mentor & Elevate Engineering: Serve as a technical authority, mentoring senior engineers and driving high engineering standards across training stability, reproducibility, and system design

Benefits

• Medical, dental, and vision insurance for employees and dependents
• Employee assistance programs focused on mental health
• Personalized workplace adjustments for diverse needs and abilities, including physical, mental, and neurodivergent considerations
• Access to mental health apps
• Onsite wellness centers
• Second medical opinion for you and your loved ones
• Counseling services
• Support programs including menopause benefit
• Medical advocacy program for transgender employees
• Competitive compensation
• Regular bonus and equity performance grant opportunities
• Generous 401(k) and regional retirement plans
• Annual cross-company compensation review and pay equity analysis
• 1-on-1 financial coaching
• Fertility and growing family assistance
• Parental leave and baby bonding leave
• Elder care and support
• Survivor income benefit
• Backup childcare
• Caregiver leave
• Paid time off, including vacation, bereavement, sick leave, parental leave, disability, and holidays
• Jury duty leave
• Military leave
• Hybrid work model with remote work opportunities also available
• Educational reimbursement
• Peer learning and coaching platform
• Donation matching programs
• Volunteer hours
• Employee resource groups
• Internal community groups and local culture clubs
• Inspiring spaces to work, recharge, and collaborate with fellow Waymonauts
• On-site meals and snacks
• Fitness centers, massage programs, and ergonomic support
• On-demand fitness, wellbeing, and cooking classes
• Commuter benefits

Post-Training & Alignment Expertise

Hands-on experience designing post-training and student-conversion pipelines for generative models—including few-step diffusion/flow distillation (e.g., DMD, MMD, Consistency Models, Rectified Flow), autoregressive exposure-bias mitigation (e.g., Self-Forcing), SFT, and Reinforcement Learning (RLAIF / reward-model alignment)—to control and stabilize long-horizon rollouts

Deep 3D/4D Vision Understanding

Deep technical foundation in 3D computer vision and physical world modeling—including action-conditioned scene dynamics, explicit 3D physical/geometric priors (4D occupancy, dynamic meshes, rigid/articulated kinematics), spatiotemporal tokenization (3D/4D causal VAEs), continuous ray/camera geometry (e.g., Plücker rays, cross-rig calibration), and generative 4D world representations (coupling explicit physical priors with neural generation)

Large-Scale Pre-Training Track Record

Demonstrated experience training large-scale generative and physical world models from scratch (e.g., action-conditioned 3D/4D world models, large video diffusion models, autoregressive spatiotemporal transformers, flow matching, or vision-language-action models) on high-performance compute clusters (multi-node GPU/TPU)

Education

PhD or MS in Computer Science, Machine Learning, Robotics, Computer Vision, or a related quantitative field (or equivalent practical research/industry tenure, including academic appointments looking to transition to industry)

Framework Proficiency

Expert-level proficiency in Python and modern distributed ML frameworks (such as JAX/Flax, XLA/Pallas, PyTorch, Megatron-LM, Pathways, or DeepSpeed) for multi-node model, sequence, and data parallelism

Senior Technical Leadership

8+ years of industry or relevant research lab experience (or tenure-track/associate professorship equivalent) developing, deploying, and leading complex ML architectures

High-Performance Infrastructure

Experience with low-level systems optimization, XLA/Pallas TPU kernels or custom CUDA/Triton kernels, block-sparse / Flash / Splash attention over 100k+ token sequences, and high-throughput KV-cached distributed serving

Domain Pedigree

Prior experience from leading video generation labs, foundation model startups, advanced robotics institutes, or high-tier academic research groups focusing on physical AI, foundation video models, or world simulation

Multi-Modal Sensor Simulation

Background in generative modeling and synthesis of non-RGB physical sensors—specifically 3D LiDAR point clouds/range images and 4D imaging radar—alongside multi-camera automotive rigs across diverse vehicle platforms, weather, and lighting conditions

Embodied AI & World Dynamics

Hands-on experience in physical world modeling and embodied dynamics—including action-conditioned causal world models, differentiable or hybrid physics engines, rigid/articulated body kinematics, vehicle dynamics, interactive multi-agent traffic dynamics, or closed-loop robotics simulators

Systems Engineering

Working proficiency in modern C++ for high-performance simulation integration