Helix AI Engineer, Training Performance

Figure; Sacramento (CA); On-site; USD 200,000 - 400,000; Full time

Job summary

Figure, an AI robotics company, seeks an experienced AI Training Performance Engineer to optimize distributed training and kernel performance for massive models. The role focuses on GPU kernel optimization, accelerator selection, and scalable data pipelines in a high‑scale cluster environment.

You will build tooling for performance monitoring, collaborate with researchers on model architectures, and advance compiler and optimization techniques across diverse hardware, including non‑NVIDIA

Qualifications

• Bachelor's or Master's degree in CS, CE, or a related field.
• 3+ years in AI performance engineering leading large-scale projects.
• Deep understanding of GPU architecture and performance.
• Proficiency with profiling tools and translating traces into optimizations.
• Strong Python and CUDA/C++ skills; framework internals experience.
• Experience debugging performance regressions at scale.
• Experience with hardware-efficiency metrics (MFU/HFU).

Responsibilities

• Optimize training performance for 100B+ models across 100k+ GPUs.
• Collaborate on accelerator choice, cluster topology, scheduling, and hardware procurement decisions.
• Write and optimize custom kernels (Triton/CUDA).
• Build tooling and dashboards for continuous performance monitoring, regression detection, and root‑cause analysis.
• Optimize data loading and preprocessing pipelines so I/O never gates the accelerators.
• Improve checkpointing, fault tolerance, and elastic restart for large jobs.
• Partner with researchers to co‑design scalable architectures and training recipes.
• Extend kernel compilers to improve iteration speed and support non‑NVIDIA accelerators.

Skills

Python

CUDA/C++

Profiling tools

GPU architecture

Distributed training

Education

Bachelor's or Master's in CS/EE

Tools

Nsight Systems

PyTorch Profiler

HTA

NCCL

Job description

Figure is an AI robotics company developing autonomous general-purpose humanoid robots. The goal of the company is to ship humanoid robots with human level intelligence. Its robots are engineered to perform a variety of tasks in the home and commercial markets. Figure is headquartered in San Jose, CA.

Figure's vision is to deploy autonomous humanoids at a global scale. Our Helix team is looking for an experienced AI Training Performance Engineer to take our model training to the next level. This role is focused on improving distributed training frameworks for large scale model training, optimizing GPU kernels, exploring the relative gains of different accelerator types and co‑designing our models to maximize utilization of our hardware.

Responsibilities

Optimize training performance for a 100B+ parameter models across 100k+ GPUs.Collaborate with the broader team on accelerator choice, cluster topology, scheduling, and hardware procurement decisions to inform future scaling.Write and optimize custom kernels (Triton/CUDA)Build tooling and dashboards for continuous performance monitoring, regression detection, and root‑cause analysis across training jobsOptimize data loading and preprocessing pipelines so I/O never gates the acceleratorsImprove checkpointing, fault tolerance, and elastic restart so large jobs recover quickly from node failures without losing significant wall‑clock timePartner with researchers to co‑design model architectures and training recipes that are performant at scale (e.g., activation checkpointing strategies, mixed precision, sequence packing)Extend and contribute to kernel compilers (e.g., Triton, Gluon) to improve iteration speed and enable targeting of custom/non‑NVIDIA acceleratorsBuild and extend agentic systems that automatically generate, benchmark, and iterate on custom kernelsEvaluate emerging accelerator architectures (AMD, TPU, SRAM‑based ASICs, and other novel hardware) for fit with our training workloads, and lead proof‑of‑concept ports/benchmarksExplore different model/data parallelisms (FSDP, context parallel, expert parallel, etc.) to determine optimal configuration per model size.

Requirements

Bachelor's or Master's degree in Computer Science, Computer/Electrical Engineering, or a related field3+ years in AI performance engineering, with significant time leading large-scale performance improvement projectsDeep understanding of GPU architecture and performance characteristics (memory bandwidth, compute‑bound vs. memory‑bound ops, occupancy)Proficiency with profiling tools (Nsight Systems/Compute, PyTorch Profiler, HTA, or similar) and ability to translate traces into concrete optimizationsSolid grasp of collective communication (NCCL) and modern networking concepts (RDMA, NVLink, InfiniBand/RoCE, topology‑aware placement).Strong Python and CUDA/C++ skills; comfortable reading and modifying framework internalsExperience debugging performance regressions and instability at scale (stragglers, hangs, OOMs, numerical divergence)Experience defining and reasoning about hardware‑efficiency metrics (MFU/HFU) and using them to drive optimization priorities

Bonus Qualifications

Experience with heterogeneous or multi‑datacenter training setups and cross‑cluster orchestrationContributions to open‑source ML systems projects (PyTorch, Megatron‑LM, vLLM, DeepSpeed, JAX, etc.)Exposure to non‑NVIDIA accelerators (AMD GPUs, TPU/Trainium/Inferentia, or custom silicon) and heterogeneous fleet management.

The US base salary range for this full‑time position is between $200,000 - $400,000 annually.The pay offered for this position may vary based on several individual factors, including job‑related knowledge, skills, and experience. The total compensation package may also include additional components/benefits depending on the specific role. This information will be shared if an employment offer is extended.