Senior ML Inference Systems Engineer

Annapurna Labs (U.S.) Inc.; Seattle (WA); On-site; USD 180,000 - 260,000; Full time

Job summary

Amazon AWS Neuron is the complete software stack for Inferentia and Trainium, delivering high-performance model inference for customer workloads on AWS. This senior software engineering role focuses on building and optimizing large-scale inference solutions within the ML Inference Applications team.

The engineer will lead core serving tech within open-source frameworks, drive performance across frameworks like vLLM and SGLang, and collaborate with model development, compiler, runtime, and

Qualifications

• 5+ years of non-internship professional software development experience.
• 5+ years of programming with at least one software programming language experience.
• 5+ years of leading design or architecture of new and existing systems.
• 5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience.
• Experience as a mentor, tech lead or leading an engineering team.
• Fundamentals of Machine learning models, their architecture, training and inference lifecycles along with work experience on some optimizations for improving the model performance.

Responsibilities

• Lead the customization and optimization of open-source inference frameworks (vLLM, SGLang) on AWS Neuron— including core framework logic such as scheduling and model execution— applying state-of-the-art kernel development, parallel computation, distributed KV cache, speculative decoding, and systems engineering to deliver best-in-class LLM serving performance.
• Influence the team's technical roadmap by evaluating emerging inference research and translating it into production-ready features, and raise the bar through design leadership and mentorship.
• Collaborate across model development, compiler, runtime, and performance engineering teams to ensure end-to-end model performance and production-ready accuracy, scalability, and efficiency.

Skills

5+ years professional software

Mentor / tech lead

ML fundamentals

CS fundamentals

Full software lifecycle

Education

Bachelor's degree in CS or equivalent

Tools

PyTorch

JAX

CUDA

Triton

Job description

Amazon AWS Neuron is the complete software stack for Inferentia and Trainium, delivering high-performance model inference for customer workloads on AWS. This senior software engineering role focuses on building and optimizing large-scale inference solutions within the ML Inference Applications team.

The engineer will lead core serving tech within open-source frameworks, drive performance across frameworks like vLLM and SGLang, and collaborate with model development, compiler, runtime, and