AI/ML Inference Engineer - Neuron & Open-Source

Amazon Inc.; Seattle (WA); On-site; USD 144,000 - 194,000; Full time

Benefits offered by this job

RSUs

Sign-on payments

Health insurance

401(k) matching

Paid time off

Parental leave

Job summary

Amazon's AWS Neuron team is building the serving technology for large-scale model inference on Neuron chips. You will contribute core features to open-source frameworks like vLLM and SGLang, optimize performance, and broaden models supported out of the box.

You\'ll collaborate with model development, performance, compiler, and runtime engineers to deliver end-to-end model performance, applying strong engineering practices to ship reliable, high-performance inference that customers depend on.

Qualifications

• 3+ years of non-internship professional software development experience
• 3+ years of non-internship design or architecture of systems experience
• 2+ years of designing and developing large-scale, multi-tiered, multi-threaded, distributed software applications
• Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field
• Knowledge of Machine Learning and LLM fundamentals, including transformer architecture, training/inference lifecycles, and optimization techniques

Responsibilities

• Contribute core serving features to open-source inference frameworks such as vLLM and SGLang — implementing and upstreaming support for continuous batching, paged attention, quantization, and distributed inference on Neuron.
• Broaden the range of models supported out of the box, and build tooling and automation that shortens the path from a new model to a production-ready deployment.
• Improve model development and shipping velocity by reducing enablement time for new models and strengthening the test, benchmarking, and release workflows the team relies on.
• Collaborate with model development, performance, compiler, and runtime engineers to deliver end-to-end model performance — production-ready accuracy, scalability, and efficiency across a broad range of models and customer workloads.
• Apply strong engineering practices — code reviews, testing, and operational excellence — to ship reliable, high-performance inference that customers depend on.

Skills

Software development

System design

Multithreaded apps

Machine Learning fundamentals

Education

Bachelor's degree in Computer Science, Engineering, Mathematics

Tools

CUDA

Triton

vLLM

SGLang

TensorRT

Job description

Amazon's AWS Neuron team is building the serving technology for large-scale model inference on Neuron chips. You will contribute core features to open-source frameworks like vLLM and SGLang, optimize performance, and broaden models supported out of the box.

You\'ll collaborate with model development, performance, compiler, and runtime engineers to deliver end-to-end model performance, applying strong engineering practices to ship reliable, high-performance inference that customers depend on.