Senior / Staff ML Training Optimization Engineer

3M HEALTHCARE; San Francisco, Phoenix, Pittsburgh, Dallas (CA, AZ, Allegheny County, TX); Hybrid; USD 140,000 - 210,000; Full time

Benefits offered by this job

Competitive compensation

Job summary

Waabi is seeking a highly skilled engineer to develop distributed training frameworks and optimize performance for research and production environments. You will profile runtime/memory, identify new technologies such as CUDA kernels and quantization, and collaborate with researchers to implement best practices for efficient resource usage.

You will also design and improve tooling and dashboards to promote broad adoption across teams, contributing to Waabi’s cutting-edge autonomous driving

Qualifications

• MS/PhD or Bachelor's with 4+ years in CS/Robotics or related field
• Strong coding skills in Python, C++, or Rust
• Experience with DL frameworks like PyTorch or JAX
• Experience profiling CPU/GPU using profiling tools
• Collaborative team player with passion for self-driving tech

Responsibilities

• Build standardized distributed training frameworks for research and production.
• Profile model runtime and memory to optimize performance.
• Evaluate emerging technologies for Waabi’s training/inference frameworks (CUDA kernels, quantization, deployment).
• Collaborate with researchers and ML engineers on best practices for resource usage.
• Create tooling and dashboards to drive broad adoption of work.

Skills

Python

C++

Rust

PyTorch

Nsight

Education

MS/PhD or Bachelor's with 4+ years experience

Tools

NVIDIA Nsight

PyTorch Profiler

CUDA

Bazel

Kubernetes

Job description

Waabi, founded by AI visionary Raquel Urtasun, is the leader in Physical AI. With a world‑class team, we’re unlocking the next era of autonomous transportation with technology that powers commercial autonomous trucks and robotaxis. Waabi is backed by and partners with world leaders in AI, automotive, logistics, and deep tech. With offices in Toronto, San Francisco, Dallas, and Pittsburgh, we are growing quickly and looking for diverse, innovative and collaborative candidates who want to impact the world in a positive way. To learn more visit: www.waabi.ai

Responsibilities

• Build standardized distributed training frameworks for research and production, drive our training towards new levels of stability and efficiency.
• Comprehensively profile model runtime and memory to pinpoint performance bottlenecks.
• Identify and evaluate emerging technologies that can be adopted into Waabi’s training and inference frameworks. Examples include designing new CUDA kernels, quantization‑aware training and inference, and compilation/deployment techniques.
• Work with researchers and ML engineers on best‑practices for optimal resource usage.
• Create and improve tooling and dashboards to ensure broad adoption of your work.

Qualifications

• MS/PhD or Bachelors degree with a minimum of 4 years of industry experience in Computer Science, Robotics and/or similar technical field(s) of study.
• Solid coding proficiency in a variety of coding languages including Python, C++ or Rust.
• Experience in deep learning frameworks such as PyTorch or Jax.
• Skilled in profiling CPU and GPU code using tools such as PyTorch Profiler and NVIDIA Nsight.
• Open‑minded and collaborative team player with willingness to help others.
• Passionate about self‑driving technologies, solving hard problems, and creating innovative solutions.

Bonus / Nice to have

• Experience in identifying when custom CUDA kernels are needed, and implementing them.
• Experience in Bazel in a monorepo environment, and integrating third‑party packages into dev environments.
• Experience with Kubernetes‑based training platforms.