Senior ML Inference Systems Engineer - Data Plane

Amazon Inc.; Cupertino (CA); On-site; USD 193,000 - 262,000; Full time

Benefits offered by this job

Health insurance

Restricted stock units (RSUs)

401(k) matching

Job summary

Amazon in Cupertino, CA is seeking a Senior Software Development Engineer to own the design and implementation of the ML inference data plane. You will optimize kernels for a custom accelerator, validate end-to-end architectures, and integrate backends into open-source frameworks for production-grade performance.

You will mentor engineers, drive architecture reviews, and raise engineering standards across the team, working across CPU, GPU, simulator, and hardware targets in a frontier-scale

Qualifications

• BS in CS or equivalent required.
• 7+ years full Software Development Life Cycle.
• Proficiency in C/C++ and Linux.
• Experience building compute kernels for GPUs/accelerators.
• Ownership of features end-to-end from design to delivery.

Responsibilities

• Develop and optimize compute kernels for a custom ML accelerator architecture, targeting production-level performance for large language model inference.
• Implement and validate LLM architectures end-to-end - from PyTorch model definition through distributed execution on custom hardware.
• Integrate custom accelerator backends into open-source ML serving frameworks (vLLM, PyTorch) including scheduler extensions, memory management, and model parallelism.
• Build and maintain test infrastructure for model correctness validation across CPU, GPU, simulator, and hardware targets.
• Profile and optimize inference workloads to reduce latency and increase throughput from simulation to hardware bringup.

Skills

C/C++

Linux

Compute kernels

LLM inference

Distributed systems

Education

Bachelor's degree in computer science or equivalent

Tools

JAX

PyTorch

vLLM

SGLang

Dynamo

TorchXLA

TensorRT

Job description

Amazon in Cupertino, CA is seeking a Senior Software Development Engineer to own the design and implementation of the ML inference data plane. You will optimize kernels for a custom accelerator, validate end-to-end architectures, and integrate backends into open-source frameworks for production-grade performance.

You will mentor engineers, drive architecture reviews, and raise engineering standards across the team, working across CPU, GPU, simulator, and hardware targets in a frontier-scale