Lead AI/ML Distributed Training Engineer on Trainium

Amazon Web Services (AWS); Cupertino (CA); On-site; USD 193,000 - 262,000; Full time

Job summary

Amazon Web Services (AWS) Annapurna Labs team seeks engineers to optimize distributed training for Trainium. You will enhance training throughput and model FLOPs utilization while collaborating across PyTorch, JAX, and the Neuron stack to push performance and scalability.

You will mentor peers and shape future architecture within a highly technical, open collaboration culture. You will work on a diverse stack from frameworks to hardware, applying novel parallelism strategies and tuning kernels

Qualifications

• 5+ years of non-internship professional software development experience.
• 5+ years of programming with at least one software programming language experience.
• 5+ years of leading design or architecture (design patterns, reliability and scaling) of new and existing systems experience.
• 5+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience.
• Experience as a mentor, tech lead or leading an engineering team.
• Experience in machine learning, data mining, information retrieval, statistics or natural language processing.

Responsibilities

• Lead efforts to optimize distributed training performance on Trainium, focusing on throughput, FLOPs utilization, and time to convergence across the Neuron stack.
• Work across PyTorch, JAX, and the Neuron compiler/runtime to enable large-scale training on Trainium instances and identify missing operators and sharding strategies.
• Own parallelism strategies like data, tensor, pipeline, expert, and context parallelism, applying reduced-precision formats when beneficial.
• Profile end-to-end workloads to determine bottlenecks and drive fixes with compiler, runtime, and collectives engineers.
• Translate performance gaps into requirements influencing future Trainium architecture and contribute upstream to open-source frameworks.

Skills

Programming experience

Leadership experience

Mentor

ML experience

SDLC experience

Education

Bachelor's degree in computer science or equivalent

Tools

PyTorch/JAX/TensorFlow

Job description

Amazon Web Services (AWS) Annapurna Labs team seeks engineers to optimize distributed training for Trainium. You will enhance training throughput and model FLOPs utilization while collaborating across PyTorch, JAX, and the Neuron stack to push performance and scalability.

You will mentor peers and shape future architecture within a highly technical, open collaboration culture. You will work on a diverse stack from frameworks to hardware, applying novel parallelism strategies and tuning kernels