ML Inference Systems Engineer — High-Performance Serving

Annapurna Labs (U.S.) Inc.; Seattle (WA); On-site; USD 144,000 - 194,000; Full time

Benefits offered by this job

RSUs

401(k) matching

Paid time off

Job summary

Amazon in Seattle is seeking engineers to advance the Neuron-based serving stack, enabling fast, scalable model inference across a wide range of models. You will contribute core features to open-source frameworks, optimize performance on Neuron hardware, and broaden model support with robust tooling and validation.

The role emphasizes collaboration across model development, performance, and runtime teams, with a focus on end-to-end efficiency, reliability, and release readiness in production

Qualifications

• 3+ years of non-internship professional software development experience
• 3+ years of design or architecture experience for large-scale systems
• 2+ years designing multi-tiered, distributed software applications
• Bachelor's degree or foreign equivalent in CS/Engineering/Math or related field
• Knowledge of ML and LLM fundamentals, transformer architectures, and optimization techniques

Responsibilities

• Contribute core serving features to open-source inference frameworks such as vLLM and SGLang, implementing and upstreaming support for continuous batching, paged attention, quantization, and distributed inference on Neuron
• Broaden the range of models supported out of the box and build tooling that shortens the path from a new model to production deployment
• Improve model development and shipping velocity by reducing enablement time and strengthening test, benchmarking, and release workflows
• Collaborate with model development, performance, compiler, and runtime engineers to deliver end-to-end model performance across a broad range of models and workloads
• Apply strong engineering practices — code reviews, testing, and operational excellence — to ship reliable, high-performance inference

Skills

C++

Java

C#

Perl

Distributed systems

Machine learning basics

Education

Bachelor's degree in Computer Science or related field

Tools

vLLM

SGLang

TensorRT

CUDA

Triton

Job description

Amazon in Seattle is seeking engineers to advance the Neuron-based serving stack, enabling fast, scalable model inference across a wide range of models. You will contribute core features to open-source frameworks, optimize performance on Neuron hardware, and broaden model support with robust tooling and validation.

The role emphasizes collaboration across model development, performance, and runtime teams, with a focus on end-to-end efficiency, reliability, and release readiness in production