ML Performance Engineer: Scale Training & Throughput

Applied Intuition; Sunnyvale (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Applied Intuition in Sunnyvale is seeking a Performance Engineer to accelerate large-scale ML workloads in data centers. You will optimize distributed training across many nodes and improve batch inference on petabyte-scale sensor logs, targeting throughput and cost-per-data processed.

You will own profiling across the stack, from data loading to kernel execution, and work with accelerators, ML frameworks, and data infrastructure to close performance gaps and drive faster iterations.

Qualifications

• Hands-on ML performance engineering experience: profiling, roofline analysis, throughput optimization, root-cause investigation.
• Experience with distributed multi-node training at scale (FSDP, DeepSpeed, Megatron, NCCL, or equivalent).
• Deep familiarity with GPU or accelerator performance concepts - memory bandwidth, kernel launch overhead, occupancy, quantization, collective communication.
• Experience with high-throughput or batch inference systems (NVIDIA Triton Inference Server, TensorRT, ONNX Runtime, Ray, or similar).
• Fluency in Python and proficiency in C++ or another systems language.
• Excellent debugging, analytical, and problem-solving skills.
• Deep understanding of machine learning foundations and ability to develop solutions for problems without established playbooks.

Responsibilities

• Profile and optimize distributed training end to end across data loading, preprocessing, kernel execution, and checkpointing.
• Optimize large-scale offline and batch inference over petabyte-scale sensor logs.
• Establish roofline and performance models and rank optimization opportunities by impact.
• Improve multi-node scaling efficiency through sharding, parallelism, and memory/bandwidth optimization.
• Drive cluster goodput by reducing GPU idle time due to I/O, scheduling gaps, or stragglers.
• Build benchmarking, observability, and regression-detection tooling to protect performance as models evolve.
• Collaborate with engineers across functions to solve data and compute problems at scale.
• Contribute to a culture of collaboration, technical excellence, and innovation.

Skills

Profiling & roofline analysis

Distributed multi-node training

Python

C++

GPU / accelerator knowledge

Troubleshooting & debugging

Tools

NVIDIA Triton Inference Server

TensorRT

ONNX Runtime

Kubernetes

Ray

Job description

Applied Intuition in Sunnyvale is seeking a Performance Engineer to accelerate large-scale ML workloads in data centers. You will optimize distributed training across many nodes and improve batch inference on petabyte-scale sensor logs, targeting throughput and cost-per-data processed.

You will own profiling across the stack, from data loading to kernel execution, and work with accelerators, ML frameworks, and data infrastructure to close performance gaps and drive faster iterations.