S Senior Staff Machine Learning Performance Engineer Santa Clara, CA, US Principal Python C++ PyTorch

ContentBuffer

Santa Clara (CA)

On-site

USD 150,000 - 210,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

XPENG is seeking a Senior Staff Machine Learning Performance Engineer to own the speed of end-to-end autonomous model training across thousands of accelerators. You will optimize GPU training and inference workloads in production, diagnose bottlenecks, and push fixes to become default implementations.

You will collaborate with modeling, infrastructure, simulation, data-mining and robotics teams to drive scalable performance improvements and robust, auditable systems.

Qualifications

  • 8+ years of industry experience in performance engineering.
  • Hands-on experience optimizing GPU training or inference workloads in production at multi-node scale.
  • Proven track record diagnosing performance problems in production workloads and delivering fixes.

Responsibilities

  • Train performance at scale: optimize step time, scaling, and long-run stability across accelerators.
  • Lead cross-team technical initiatives with modeling, infrastructure, simulation, data-mining and robotics teams.
  • Create reusable components that become defaults across models and workloads.
  • Extend observability tooling and metrics with low instrumentation overhead for production jobs fleet-wide.
  • Build agentic automation: reproduce jobs on demand and verify numerical parity to catch regressions.

Skills

Advanced performance engineering
GPU training optimization
Production systems debugging

Education

Master's or PhD in CS, CE or EE

Job description

XPENG is a leading smart technology company at the forefront of innovation, integrating advanced AI and autonomous driving technologies into its vehicles, including electric vehicles (EVs), electric vertical take-off and landing (eVTOL) aircraft, and robotics. With a strong focus on intelligent mobility, XPENG is dedicated to reshaping the future of transportation through cutting-edge R&D in AI, machine learning, and smart connectivity.

The Mission:

We are building the next generation of end-to-end autonomous driving on physical AI foundation models — vision-language-action policies and world models — and the same training stack carries our robotics program. These training runs span thousands of accelerators and days of wall-clock — how fast we train them is how fast we learn. As a Senior Staff Machine Learning Performance Engineer, you will own that speed: working with the modeling teams on their hardest bottlenecks, setting the technical direction for performance, and turning one-off wins into defaults — including the agentic workflows that find the next round of them automatically.

Job Responsibilities:
  • Training Performance at Scale: Own step time, scaling efficiency and long-run stability for foundation-model training across thousands of accelerators — from the data path to kernels, parallelism and communication.
  • Cross-Team Technical Leadership: Work directly with the modeling, infrastructure, simulation, data-mining and robotics teams on their hardest bottlenecks, and set the performance direction those teams build against.
  • Reusable by Design: Turn one-off wins into modular components that become defaults across most models and workloads, rather than a patch that helps a single job.
  • Observability at Scale: Extend performance tooling and metric coverage across the stack, while keeping instrumentation overhead low enough to stay on for production jobs fleet-wide.
  • Agentic Automation: Build agents that reproduce any job on demand, verify numerical parity and separate a real regression from run-to-run noise — so that most of the optimizations that land are found and fixed by AI rather than by hand.
Minimum Skill Requirements:
  • Master's or PhD in CS, CE or EE, or equivalent practical experience, with 8+ years of industry experience, the majority of it in performance engineering.
  • Hands-on experience optimizing GPU training or inference workloads in production, at multi-node scale.
  • A track record of diagnosing performance problems in real production workloads and landing the fixes — often in systems owned by other teams — with the gain measured and verified in production.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Staff Machine Learning Performance Engineer
Senior Staff Machine Learning Performance Engineer

Xpeng Inc. • Santa Clara (CA)

On-site
USD 244,000 - 413,000
Competitive compensation package
Snacks, lunches, dinners, and funกิจ
Senior Staff Machine Learning Performance Engineer
Senior Staff Machine Learning Performance Engineer

Xpengmotors • Santa Clara (CA)

On-site
USD 244,000 - 413,000
Competitive compensation package
Snacks, lunches, and dinners
Senior ML Performance Engineer – Scaled Training
Senior ML Performance Engineer – Scaled Training

Xpengmotors • Santa Clara (CA)

On-site
USD 244,000 - 413,000
Competitive compensation package
Snacks, lunches, and dinners
Senior ML Performance Engineer – Scale Training & Systems
Senior ML Performance Engineer – Scale Training & Systems

ContentBuffer • Santa Clara (CA)

On-site
USD 150,000 - 210,000
Senior ML Performance Engineer - Scale GPU Training
Senior ML Performance Engineer - Scale GPU Training

Xpeng Inc. • Santa Clara (CA)

On-site
USD 244,000 - 413,000
Competitive compensation package
Snacks, lunches, dinners, and funกิจ
Senior Machine Learning Engineer - AI Foundation
Senior Machine Learning Engineer - AI Foundation

Xpeng Inc. • Santa Clara (CA)

On-site
USD 175,000 - 296,000
Snacks
Lunches
Dinners
+1
Staff Machine Learning Engineer - Foundation Model
Staff Machine Learning Engineer - Foundation Model

XPENG • Santa Clara (CA)

On-site
USD 215,280 - 364,320
Competitive compensation package
Snacks, lunches, dinners, and fun activities
Access to massive real-world data and industry-scale compute
Senior Machine Learning Engineer - Foundation Model
Senior Machine Learning Engineer - Foundation Model

Xpeng Inc. • Santa Clara (CA)

On-site
USD 175,000 - 296,000
Real-world data access
Top-tier research team
Impact on mobility
+3
GPGPU Software Architect/ Principal Engineer
GPGPU Software Architect/ Principal Engineer

Xpeng Inc. • Santa Clara (CA)

On-site
USD 241,800 - 409,200
Medical insurance
Vision insurance
401(k)
Staff Software Engineer - Scalable AI/ML Infra for Autonomy
Staff Software Engineer - Scalable AI/ML Infra for Autonomy

Top AI's Jobs • United States

Hybrid
USD 179,000 - 304,000