Senior Principal AI Inference Systems Architect

Ampere; Santa Clara (CA); On-site; USD 237,000 - 356,000; Full time

Benefits offered by this job

Health insurance

401K retirement plan

Paid holidays

Job summary

Ampere is seeking a Sr. Principal Machine Learning Software Engineer to lead architecture and development of high-performance AI inference-serving software for Ampere accelerators.

You will work across model serving, frameworks, compilers, runtimes, and optimized kernels to enable efficient deployment of modern AI workloads, including generative AI and large language models. This is a hands-on technical leadership role for someone who can solve complex systems and performance challenges while

Qualifications

• Proven experience building or optimizing production inference runtimes or serving engines.
• Deep understanding of batching, scheduling, latency/throughput tradeoffs.
• Strong systems-level programming and memory management skills.
• Experience with profiling, benchmarking and low-level debugging.

Responsibilities

• Architect and lead an inference-serving engine on Ampere accelerators.
• Drive end-to-end serving performance, throughput and latency targets.
• Collaborate with model-serving, framework, compiler, and hardware teams.
• Implement batching, KV-cache, memory planning, and quantization.
• Shape hardware/software co-design directions for future accelerators.
• Mentor engineers and drive cross-functional technical initiatives.
• Evaluate open-source AI serving tools and integrate where appropriate.
• Partner with product and customer teams to prioritize capabilities.

Skills

C++ expert

Performance optimization

AI inference

Systems knowledge

Profiling/debugging

Cross-functional leadership

Memory management

Concurrency

KV-cache knowledge

Education

Bachelor’s in CS/CE/Math/EE

Master’s degree (advantage)

PhD (advantage)

Tools

CUDA

PyTorch

ONNX Runtime

TensorRT

MLIR

TVM

XLA

OpenVINO

Kubernetes

Job description

Ampere is seeking a Sr. Principal Machine Learning Software Engineer to lead architecture and development of high-performance AI inference-serving software for Ampere accelerators.

You will work across model serving, frameworks, compilers, runtimes, and optimized kernels to enable efficient deployment of modern AI workloads, including generative AI and large language models. This is a hands-on technical leadership role for someone who can solve complex systems and performance challenges while