ML Inference Engineer II — Optimize LLMs at Scale

Amazon Inc.; Santa Clara (CA); On-site; USD 165,000 - 224,000; Full time

Job summary

Amazon AI is seeking a Machine Learning Engineer II to join the Inference Optimization team in Santa Clara, CA. You will drive price-performance for LLM inference on SageMaker, Bedrock and Hyperpod, building model servers and tuning CUDA kernels for large language models.

You will work cross-functionally with researchers and engineers to deploy production-ready inference optimizations, evaluate new techniques like speculative decoding and disaggregated inference, and contribute to scalable,

Qualifications

• 3+ years of non-internship software development experience
• 2+ years of non-internship design/architecture experience for large systems
• 1+ years of software development engineer or related experience
• 1+ years of designing and developing large‑scale, multi‑tiered software using C#, C++, Java, or Perl
• Bachelor's degree in Computer Science, Engineering, Mathematics or related field
• Knowledge of ML and LLM fundamentals including transformers, training/inference lifecycles, and optimization techniques

Responsibilities

• Develop model servers and CUDA kernels in vLLM and SGLang
• Develop services for GenAI optimization and benchmarking
• Create and modify a mid-size application or service
• Own price-performance of LLM inference on SageMaker, Hyperpod and Bedrock

Skills

C++

C#

Java

Perl

Machine Learning

Education

Bachelor's degree in Computer Science / Engineering / Mathematics

Tools

vLLM

SGLang

CUDA

Job description

Amazon AI is seeking a Machine Learning Engineer II to join the Inference Optimization team in Santa Clara, CA. You will drive price-performance for LLM inference on SageMaker, Bedrock and Hyperpod, building model servers and tuning CUDA kernels for large language models.

You will work cross-functionally with researchers and engineers to deploy production-ready inference optimizations, evaluate new techniques like speculative decoding and disaggregated inference, and contribute to scalable,