ML Inference Engineer - Custom Hardware Data Plane

Socket.dev; Cupertino (CA); On-site; USD 165,000 - 224,000; Full time

Benefits offered by this job

RSUs

Health benefits

401(k) matching

Job summary

Amazon in Cupertino, CA is seeking a Software Development Engineer to own the design and implementation of our inference data plane, building software that runs large models efficiently on custom hardware.

You will develop high-performance compute kernels, work across the inference path from model definition to distributed execution, and contribute to CI/CD, testing, and profiling infrastructure to push latency and throughput at frontier scale.

Qualifications

• Bachelor's degree or equivalent.
• 4+ years of full software development life cycle experience.
• Knowledge of computer architecture, operating systems, and parallel computing.
• Knowledge of Linux fundamentals.
• Strong proficiency in C/C++.
• Experience developing compute kernels for GPUs, DSPs, or custom accelerators.
• Proven track record of owning and delivering complex software features end-to-end.

Responsibilities

• Develop and optimize compute kernels for a custom ML accelerator architecture, targeting production-level performance for large language model inference.
• Implement and validate LLM architectures end-to-end — from PyTorch model definition through distributed execution on custom hardware.
• Integrate custom accelerator backends into open-source ML serving frameworks (vLLM, PyTorch), including scheduler extensions, memory management, and model parallelism.
• Build and maintain test infrastructure for model correctness validation across CPU, GPU, simulator, and hardware targets.
• Profile and optimize inference workloads — identify bottlenecks, instrument critical paths, and drive latency and throughput improvements from simulation through hardware bring-up.
• Own features end-to-end: from design through implementation, testing, and integration into the broader software stack.
• Contribute to CI/CD pipelines that gate model and kernel changes on correctness and performance regressions.

Skills

C/C++

Linux fundamentals

Parallel computing

Computer architecture

OS knowledge

Education

Bachelor's degree or equivalent

Tools

JAX

PyTorch

vLLM

SGLang

Dynamo

TorchXLA

TensorRT

Job description

Amazon in Cupertino, CA is seeking a Software Development Engineer to own the design and implementation of our inference data plane, building software that runs large models efficiently on custom hardware.

You will develop high-performance compute kernels, work across the inference path from model definition to distributed execution, and contribute to CI/CD, testing, and profiling infrastructure to push latency and throughput at frontier scale.