Senior AI/ML Inference Engineer for Neuron SDK

Amazon; Cupertino (CA); On-site; USD 193,000 - 262,000; Full time

Benefits offered by this job

Health insurance

RSU program

Job summary

Amazon is seeking a Senior Software Development Engineer for AI/ML in the AWS Neuron team, focused on model inference and accelerator optimization. You will lead distributed inference support for PyTorch in the Neuron SDK, tune models for latency and throughput on Trainium and Inferentia, and collaborate across compiler, runtime, framework, and hardware groups to maximize performance.

You will work with customers to enable model deployment on AWS accelerators, contribute to open-source

Qualifications

• Bachelor's degree in computer science or equivalent.
• 5+ years of non-internship professional software development experience.
• Fundamentals of Machine learning and LLMs, their architecture, training and inference lifecycles.

Responsibilities

• Design, develop, and optimize machine learning models and frameworks for deployment on custom ML hardware accelerators.
• Participate in all stages of the ML system development lifecycle including distributed computing based architecture design, implementation, performance profiling, hardware-specific optimizations, testing and production deployment.
• Build infrastructure to systematically analyze and onboard multiple models with diverse architecture.
• Design and implement high-performance kernels and features for ML operations, leveraging the Neuron architecture and programming models
• Analyze and optimize system-level performance across multiple generations of Neuron hardware
• Conduct detailed performance analysis using profiling tools to identify and resolve bottlenecks
• Implement optimizations such as fusion, sharding, tiling, and scheduling
• Conduct comprehensive testing, including unit and end-to-end model testing with continuous deployment and releases through pipelines.
• Work directly with customers to enable and optimize their ML models on AWS accelerators
• Collaborate across teams to develop innovative optimization techniques

Skills

C++

Python

System performance

Debugging & profiling

Parallel computing

ML concepts

PyTorch familiarity

Education

Bachelor's degree in computer science or equivalent

Tools

CUDA kernels

TensorRT

vLLM

SGLang

JIT compilation

Job description

Amazon is seeking a Senior Software Development Engineer for AI/ML in the AWS Neuron team, focused on model inference and accelerator optimization. You will lead distributed inference support for PyTorch in the Neuron SDK, tune models for latency and throughput on Trainium and Inferentia, and collaborate across compiler, runtime, framework, and hardware groups to maximize performance.

You will work with customers to enable model deployment on AWS accelerators, contribute to open-source