Senior ML Inference Systems Engineer - Custom Accelerator

Amazon; Cupertino (CA); On-site; USD 193,000 - 262,000; Full time

Benefits offered by this job

Health insurance

RSUs

401(k) matching

Job summary

Amazon is seeking a Senior Software Development Engineer to own the design and implementation of the inference data plane for large models on custom hardware. You will build high-performance kernels, validate architectures end-to-end, and drive optimizations across CPU, GPU, and hardware targets.

You will integrate backends into ML serving frameworks, contribute to CI/CD, and mentor engineers while advancing the frontier of model inference at scale in a ground-up environment.

Qualifications

• Bachelor's degree in computer science or equivalent.
• 7+ years of full software development life cycle experience.
• Strong proficiency in C/C++.
• Strong Linux systems knowledge.
• Experience developing compute kernels for GPUs, DSPs, or custom accelerators.
• Proven track record of owning and delivering complex software features end-to-end.

Responsibilities

• Develop and optimize compute kernels for a custom ML accelerator architecture.
• Implement and validate LLM architectures end-to-end—from PyTorch model definition to distributed execution on custom hardware.
• Integrate custom accelerator backends into open-source ML serving frameworks (vLLM, PyTorch).
• Build and maintain test infrastructure for model correctness across CPU, GPU, simulator, and hardware targets.
• Profile and optimize inference workloads to reduce latency and increase throughput across the stack.
• Own features end-to-end from design through implementation, testing, and integration.
• Contribute to CI/CD pipelines ensuring correctness and performance.
• Mentor engineers and elevate engineering standards across the team.

Skills

C/C++

Linux

Compute kernels

End-to-end ownership

Model serving

Distributed systems

Education

B.S. CS or equivalent

Tools

PyTorch

vLLM

JAX

TensorRT

TorchXLA

Dynamo

Job description

Amazon is seeking a Senior Software Development Engineer to own the design and implementation of the inference data plane for large models on custom hardware. You will build high-performance kernels, validate architectures end-to-end, and drive optimizations across CPU, GPU, and hardware targets.

You will integrate backends into ML serving frameworks, contribute to CI/CD, and mentor engineers while advancing the frontier of model inference at scale in a ground-up environment.