ML Inference Engineer II: AWS & SageMaker Optimization

Amazon Development Center U.S., Inc.; Santa Clara (CA); On-site; USD 165,000 - 224,000; Full time

Benefits offered by this job

Sign-on payments

Restricted stock units (RSUs)

Health insurance (medical, dental, and

Vision, prescription

401(k) matching

Paid time off

Parental leave

Workplace accommodation during process

Job summary

Amazon Development Center U.S., Inc. is seeking a Machine Learning Engineer focused on AWS inference optimization in Santa Clara, CA. You will own price-performance for LLM inference across SageMaker, Bedrock, and Hyperpod, and drive performance improvements for large language OSS models.

You will develop model servers, build CUDA kernels, and create services for GenAI optimization and benchmarking, delivering production-ready solutions integrated with customer-facing products.

Qualifications

• 3+ years of non-internship software development experience.
• 2+ years of non-internship design or architecture experience, including design patterns and reliability and scaling of new and existing systems.
• 1+ year of software development engineer or related occupational experience.
• 1+ year designing and developing large-scale, multi-tiered, multi-threaded, embedded or distributed software applications, tools, systems, and services using C#, C++, Java, or Perl.
• Bachelor’s degree (or foreign equivalent) in Computer Science, Engineering, Mathematics, or a related field.
• Knowledge of machine learning and LLM fundamentals, including transformer architecture, training/inference lifecycles, and optimization techniques.

Responsibilities

• Develop model servers.
• Build CUDA kernels in vLLM and SGLang.
• Create services for GenAI optimization and benchmarking.
• Create or modify a mid-size application or service.
• Optimize large language OSS models such as Qwen, Gemma, gptoss, and others by improving inference stacks including vLLM and SGLang.
• Perform performance tuning for GenAI inference on Amazon SageMaker, Amazon Bedrock, and Hyperpod.
• Implement inference optimization techniques including Speculative decoding, Disaggregated Inference, and CUDA kernel tuning.
• Own price-performance for Amazon LLM inference deployed on SageMaker, Hyperpod, and Bedrock.
• Deliver production solutions that leverage existing academic and industrial research or apply practical, original thinking.
• Deliver solutions into customer-facing products.

Skills

C#

C++

Java

Perl

Transformer architecture

LLM fundamentals

Education

Bachelor’s degree in CS/Engineering/Math

Tools

Amazon SageMaker

Amazon Bedrock

Hyperpod

Job description

Amazon Development Center U.S., Inc. is seeking a Machine Learning Engineer focused on AWS inference optimization in Santa Clara, CA. You will own price-performance for LLM inference across SageMaker, Bedrock, and Hyperpod, and drive performance improvements for large language OSS models.

You will develop model servers, build CUDA kernels, and create services for GenAI optimization and benchmarking, delivering production-ready solutions integrated with customer-facing products.