AI Training Performance Engineer — Large-Scale GPU Training

Figure; San Jose (CA); On-site; USD 200,000 - 400,000; Full time

Job summary

Figure, an AI robotics company headquartered in San Jose, CA, is seeking an experienced AI Training Performance Engineer to advance our large-scale model training. You will optimize distributed training across thousands of GPUs, tune GPU kernels, and co-design training recipes with researchers.

You will build tooling and dashboards, improve data pipelines, and extend kernel compilers for non-NVIDIA accelerators.

Qualifications

• Degree in CS, CE, or related field required.
• 3+ years in AI performance engineering leading large-scale projects.
• Expertise in GPU architecture, memory bandwidth, and performance profiling.
• Strong Python and CUDA/C++ and ability to read/modify framework internals.
• Experience with NCCL and RDMA/InfiniBand topology-aware placement.

Responsibilities

• Optimize training performance for 100B+ parameter models across 100k+ GPUs.
• Collaborate on accelerator choice, cluster topology, scheduling, and hardware procurement.
• Write and optimize custom kernels (Triton/CUDA).
• Build tooling and dashboards for performance monitoring and root-cause analysis across training jobs.
• Optimize data loading and preprocessing pipelines to avoid I/O bottlenecks.
• Improve checkpointing, fault tolerance, and elastic restart.
• Co-design model architectures and training recipes for scale.
• Extend kernel compilers to support non-NVIDIA accelerators.
• Build agentic systems to benchmark and iterate on custom kernels.
• Evaluate emerging accelerator architectures and port benchmarks.

Skills

GPU Architecture

Profiling Tools

NCCL & Networking

Python & CUDA/C++

Debugging at scale

Hardware-efficiency metrics

Education

Bachelor's or Master's degree in CS/CE or related field

Tools

Nsight Systems

PyTorch Profiler

HTA

Job description

Figure, an AI robotics company headquartered in San Jose, CA, is seeking an experienced AI Training Performance Engineer to advance our large-scale model training. You will optimize distributed training across thousands of GPUs, tune GPU kernels, and co-design training recipes with researchers.

You will build tooling and dashboards, improve data pipelines, and extend kernel compilers for non-NVIDIA accelerators.