Software Engineer II: AI/ML Inference on Custom Hardware

Amazon Inc.; Cupertino (CA); On-site; USD 165,000 - 224,000; Full time

Benefits offered by this job

Health insurance

RSUs

401(k) matching

Paid time off

Parental leave

Job summary

Amazon Web Services (AWS) Annapurna Labs team is hiring a Software Engineer II - AI/ML to design and implement features and own model performance on AWS Trainium end to end. You will work across ML frameworks, kernels, and runtimes to optimize inference and enable scale for GenAI workloads.

You'll collaborate with customers to enable and optimize their models on AWS accelerators, build high-performance kernels, and contribute to compiler/runtime improvements across multiple generations of Neuron

Qualifications

• 3+ years of non-internship software development experience.
• 3+ years of design or architecture experience for large systems.
• 1+ years designing and developing large-scale, distributed software applications.
• Bachelor's degree in Computer Science, Engineering, Mathematics, or a related field.
• Experience debugging, profiling, and applying software engineering best practices in large-scale systems.
• Experience with Machine Learning fundamentals, architecture, training/inference lifecycles, and optimization of model execution.
• Knowledge of Python and/or C++ programming.
• Strong understanding of system performance, memory management, and parallel computing principles.

Responsibilities

• Design, develop, and optimize ML models and frameworks for deployment on custom ML hardware accelerators.
• Participate in all stages of the ML system development lifecycle, including distributed computing-based architecture design, implementation, performance profiling, hardware-specific optimizations, testing, and production deployment.
• Build infrastructure to systematically analyze and onboard multiple models with diverse architectures.
• Understand and produce NKI (Neuron Kernel Interface) kernels, and design and implement high-performance kernels and features for ML operations, leveraging the Neuron architecture and programming models.
• Analyze and optimize system-level performance across multiple generations of Neuron hardware, using knowledge of the underlying Trainium hardware architecture to guide optimization decisions.
• Conduct detailed performance analysis using profiling tools to identify and resolve bottlenecks.
• Implement optimizations such as fusion, sharding, tiling, and scheduling.
• Conduct comprehensive testing, including unit and end-to-end model testing with continuous deployment and releases through pipelines.
• Work directly with customers to enable and optimize their ML models on AWS accelerators.
• Collaborate across teams to develop innovative optimization techniques, and share proposals, findings, and learnings with internal developers and external customers via high-quality designs and documentation.

Skills

Software development

Design/architecture

Python programming

C++ programming

Performance optimization

ML fundamentals

Debugging & profiling

Education

Bachelor's degree in Computer Science, Engineering, Mathematics, or related field

Tools

CUDA kernels

TensorRT

PyTorch

JIT compilation

AOT tracing

CUTLASS

SGLang

vLLM

Triton

Job description

Amazon Web Services (AWS) Annapurna Labs team is hiring a Software Engineer II - AI/ML to design and implement features and own model performance on AWS Trainium end to end. You will work across ML frameworks, kernels, and runtimes to optimize inference and enable scale for GenAI workloads.

You'll collaborate with customers to enable and optimize their models on AWS accelerators, build high-performance kernels, and contribute to compiler/runtime improvements across multiple generations of Neuron