Senior ML Performance Engineer - Scale GPU Training

XPENG; Santa Clara (CA); On-site; USD 244,000 - 413,000; Full time

Benefits offered by this job

Competitive compensation package

Snacks, lunches, dinners, and funกิจ

Job summary

XPENG is seeking a Senior Staff Machine Learning Performance Engineer to own the speed and efficiency of foundation-model training across thousands of accelerators. You will work with modeling, infrastructure, simulation, data-mining, and robotics teams to push performance defaults and scalable workflows, turning wins into reusable components.

The role requires a track record of diagnosing production bottlenecks, deep computer architecture knowledge, and strong C++/Python skills.

Qualifications

• Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience.
• Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
• A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
• In-depth understanding of computer architecture and high-performance computing.
• Strong C++ and Python, with a solid command of data structures and algorithms.

Responsibilities

• Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
• Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
• Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
• Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
• Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.

Skills

C++

Python

GPU optimization

Performance profiling

Education

Master's or PhD in CS/CE/EE

Equivalent practical experience

Tools

CUDA

Triton

PyTorch

Job description

XPENG is seeking a Senior Staff Machine Learning Performance Engineer to own the speed and efficiency of foundation-model training across thousands of accelerators. You will work with modeling, infrastructure, simulation, data-mining, and robotics teams to push performance defaults and scalable workflows, turning wins into reusable components.

The role requires a track record of diagnosing production bottlenecks, deep computer architecture knowledge, and strong C++/Python skills.