Senior ML Inference Engineer - LLM Serving on Accelerators

Annapurna Labs

Seattle (WA)

On-site

USD 168,000 - 227,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k) matching
Paid time off
Parental leave

Job summary

Annapurna Labs in Seattle is seeking a senior software engineer to optimize open-source inference frameworks like vLLM and SGLang for AWS Neuron. You will lead core framework logic and collaborate with model development, compiler, and runtime teams to ensure production-ready accuracy and efficiency at scale.

The role emphasizes mentorship, design leadership, and hands-on kernel-level optimization to push the performance of large-scale inference workloads on AWS accelerators.

Qualifications

  • 5+ years of non-internship professional software development experience.
  • 5+ years of programming with at least one software programming language experience.
  • 5+ years of leading design or architecture of new and existing systems.
  • 5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience.
  • Experience as a mentor, tech lead or leading an engineering team.
  • Fundamentals of Machine learning models, their architecture, training and inference lifecycles along with work experience on some optimizations for improving the model performance.

Responsibilities

  • Lead the customization and optimization of open-source inference frameworks (vLLM, SGLang) on AWS Neuron, including core framework logic such as scheduling and model execution.
  • Apply kernel development, parallel computation, distributed KV cache, speculative decoding, and systems engineering to deliver best-in-class LLM serving performance.
  • Influence the team's technical roadmap by evaluating emerging inference research and translating it into production-ready features.
  • Mentor and provide design leadership across model development, compiler, runtime, and performance engineering teams.

Skills

Software development
System design
Leadership
Code reviews
Mentorship
ML basics

Education

Bachelor's degree in CS or equivalent

Tools

PyTorch
JAX
CUDA
Triton
vLLM
SGLang

Job description

Annapurna Labs in Seattle is seeking a senior software engineer to optimize open-source inference frameworks like vLLM and SGLang for AWS Neuron. You will lead core framework logic and collaborate with model development, compiler, and runtime teams to ensure production-ready accuracy and efficiency at scale.

The role emphasizes mentorship, design leadership, and hands-on kernel-level optimization to push the performance of large-scale inference workloads on AWS accelerators.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior ML Inference Engineer — High-Performance LLM Serving
Senior ML Inference Engineer — High-Performance LLM Serving

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 168,000 - 227,000
Health insurance
401(k) matching
Paid time off
+2
ML Inference Engineer - AWS Neuron & GenAI
ML Inference Engineer - AWS Neuron & GenAI

Amazon Web Services (AWS) • Cupertino (CA)

On-site
USD 165,000 - 224,000
Senior ML Compiler Engineer — Optimize Models for Accelerators
Senior ML Compiler Engineer — Optimize Models for Accelerators

Annapurna Labs • Cupertino (CA)

On-site
USD 193,000 - 262,000
Health insurance
401(k) matching
RSUs / Stock options
Senior AI/ML Software Engineer - High-Perf Inference
Senior AI/ML Software Engineer - High-Perf Inference

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 168,000 - 227,000
ML Inference Engineer — High-Performance LLM Serving
ML Inference Engineer — High-Performance LLM Serving

Amazon Inc. • Bellevue (WA)

On-site
USD 144,000 - 194,000
Health insurance
401(k) matching
Paid time off
+1
Senior AI/ML Inference Engineer (Neuron)
Senior AI/ML Inference Engineer (Neuron)

Amazon Web Services (AWS) • Cupertino (CA)

On-site
USD 193,000 - 262,000
Health insurance
401(k) matching
Paid time off
AI/ML Inference Engineer - Neuron & Open-Source
AI/ML Inference Engineer - Neuron & Open-Source

Amazon Inc. • Seattle (WA)

On-site
USD 144,000 - 194,000
RSUs
Sign-on payments
Health insurance
+3
Software Engineer I: ML Inference & Accelerator Optimization
Software Engineer I: ML Inference & Accelerator Optimization

Amazon Web Services • Cupertino (CA)

On-site
USD 127,000 - 185,000
Health insurance
RSUs
401(k) matching
+2
Senior ML Compiler Engineer — Neuron SDK
Senior ML Compiler Engineer — Neuron SDK

Annapurna Labs (U.S.) Inc. • Cupertino (CA)

On-site
USD 180,000 - 240,000
Senior SDE — Neuron Inference & LLM Optimization
Senior SDE — Neuron Inference & LLM Optimization

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 144,000 - 194,000
Health insurance
401(k) matching
Paid time off
+1