Senior ML Inference Engine Architect

Ampere Computing LLC; Santa Clara (CA); Hybrid; USD 226,000 - 339,000; Full time

Benefits offered by this job

Health insurance

401K retirement plan

Unlimited flextime and holidays

Job summary

Ampere is seeking a Sr. Principal Machine Learning Software Engineer to lead the architecture and development of a high-performance AI inference-serving engine for Ampere accelerators.

You will work across model serving, frameworks, compilers, runtimes, and optimized kernels to deploy modern AI workloads, including generative AI and LLMs. This hands-on leadership role requires solving complex systems and performance challenges and guiding the AI software roadmap.

Qualifications

• Bachelor’s degree or higher in a technical field with extensive experience in ML software.
• Experience building or optimizing production inference runtimes and serving engines.
• Strong C/C++ skills and performance optimization expertise.
• Ability to lead cross-functional initiatives and communicate with hardware/software teams.

Responsibilities

• Architect and build high-performance inference-serving engine for Ampere accelerators.
• Drive end-to-end serving performance: throughput, latency, and cost per inference.
• Implement batching, scheduling, KV-cache management, and memory planning.
• Collaborate with model-serving, framework, compiler, runtime, kernel, and hardware teams.
• Guide hardware/software co-design and contribute to open-source AI serving technologies.
• Mentor engineers and lead technical decisions across initiatives.

Skills

Modern C++

Concurrent programming

Memory management

Systems-level optimization

Leadership of cross-functional teams

Performance benchmarking

Education

Bachelor’s degree in CS/EE/Math or related; 12+ years experience; or MS 8+; PhD 5+ years

Tools

vLLM

llama.cpp

TensorRT/TensorRT-LLM

ONNX Runtime

OpenVINO

Triton Inference Server

TVM

XLA

MLIR

Job description

Ampere is seeking a Sr. Principal Machine Learning Software Engineer to lead the architecture and development of a high-performance AI inference-serving engine for Ampere accelerators.

You will work across model serving, frameworks, compilers, runtimes, and optimized kernels to deploy modern AI workloads, including generative AI and LLMs. This hands-on leadership role requires solving complex systems and performance challenges and guiding the AI software roadmap.