Sr. Systems Development Engineer, Trainium Systems

Annapurna Labs (U.S.) Inc.

San Diego (CA)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Annapurna Labs (U.S.) Inc. is seeking deeply technical engineers to own and operate software for provisioning, monitoring, and updating large-scale AI infrastructure.

You’ll build end-to-end services that enable customers to stand up, observe, and diagnose server fleets, with a strong emphasis on architecture and customer impact. You will work autonomously in a fast-moving environment within a large company, contributing to platform evolution and tooling that serves AI training workloads at

Qualifications

  • Experience leading the design, automation, deployment, and support of large-scale infrastructure.
  • Experience programming with modern languages such as Python, Ruby, Golang, Java, C++, C#, Rust.
  • Experience with Linux/Unix environments.
  • Experience with CI/CD build pipelines.

Responsibilities

  • Own the tools and systems compatible within the operator's network, systems, and infrastructure.
  • Build tools and services that enable a customer to operate these servers at scale.
  • Build services and tools for monitoring, alerting, and anomaly detection.
  • Develop troubleshooting tools and runbooks that enable rapid diagnosis and resolution of site infrastructure issues.
  • Consult with customer teams on network design and infrastructure standards, working across organizational boundaries.
  • Drive customer expansion readiness, ensuring new customer infrastructure is ready.
  • Contribute to architecture decisions for product evolution.
  • Develop and build systems that enable operating infrastructure at scale.

Skills

Large-scale infra
Programming languages
Linux/Unix
CI/CD

Job description

Have you ever wondered what it takes to build the software systems that turn a rack of AI accelerators from inert silicon into a system a customer can run workloads on? Do you want deep ownership of a service that every one of those systems depends on to boot, run, and stay healthy?

Who are we?

Annapurna Labs is the part of Amazon that designs the custom silicon behind the cloud, including Trainium for AI training, Graviton CPUs, and the AWS Nitro system. Nearly every server in AWS runs hardware we designed. We build the chips, the systems, and the software that power some of the largest AI training and inference workloads in the world.

What do we do?

We build the software that provisions, operates, updates, and monitors the servers behind AWS's largest AI training workloads, built on Trainium, Graviton, and the AWS Nitro system. We give the people who run this hardware a complete way to stand it up, watch its health, deliver new software to it, and diagnose it, all through open, standard interfaces. We own that software from design through the running system.

Why it's high-impact?

The hardware is just silicon until our software makes it provisionable, observable, and serviceable. Every system runs through what we build. The service you own is on the path for all of them, and the people who run this hardware depend on it every day.

What's the challenge?

There are many ambiguous and difficult challenges in our fast-moving space. Our services require deep system and software expertise. You need to have the ability to work within a fast moving and startup-like environment in a large company. You will identify solutions, trying ideas, given space to fail and iterate to produce products that your customers love.

What you will do?

You will be part of a team that builds services end to end, from a blank page to an operator running it in production. You work backward from what the customer needs to build tools, systems, and network for customer.

Who would succeed in this role?

Deeply technical engineers who stay close to the customer as well as the systems architecture and design. They think about the customer experience and the outcome. A person who works autonomously and dives deep into a problem to understand how things work, when to make a subtle change, and when to disrupt the status quo to achieve the right result.

Key job responsibilities
  • Own the tools and systems that are compatible within the operator's network, systems, and infrastructure.
  • Build tools and services that enable a customer to operate these servers at scale.
  • Build services and tools for monitoring, alerting, and anomaly detection.
  • Develop troubleshooting tools and runbooks that enable rapid diagnosis and resolution of site infrastructure issues.
  • Consult with customer teams on network design and infrastructure standards, working across organizational boundaries.
  • Drive customer expansion readiness, ensuring new customer infrastructure is ready.
  • Contribute to architecture decisions for product evolution.
  • Develop and build systems that enable operating infrastructure at scale.
A day in the life

You design and build the provisioning and management surface over standard interfaces, and the boot flow operator tools drive. You build the offline software-update service and prove the signed bundle applies and recovers cleanly. You design each system against the interfaces the platform exposes and turn architecture decisions into APIs and tools a customer runs without you in the loop.

About the team

Our team builds the software suite for provisioning, monitoring, software delivery, and the diagnostics of Trainium systems, and we build all of it on open, standard interfaces. We serve the engineers and operators who run large-scale AI training and need to provision, monitor, update, and troubleshoot it on their own terms. We own it end to end. We're a small team with a large charter. You own real systems on day one. There are no layers between you and the problem. You build it, you ship it, you operate it, you improve it.

Basic Qualifications:
  • Experience leading the design, automation, deployment, and support of large-scale infrastructure
  • Experience programming with at least one modern language such as Python, Ruby, Golang, Java, C++, C#, Rust
  • Experience with Linux/Unix
  • Experience with CI/CD pipelines build processes
Preferred Qualifications:
  • Experience with distributed systems at scale

Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.

Los Angeles County applicants: Job duties for this position include: work safely and cooperatively with other employees, supervisors, and staff; adhere to standards of excellence despite stressful conditions; communicate effective ly and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and follow all federal, state, and local laws and Comp

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Amazon Web Services • Cupertino (CA)

On-site
USD 165,000 - 224,000
Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Annapurna Labs • Seattle (WA)

On-site
USD 144,000 - 194,000
Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Annapurna Labs • Cupertino (CA)

On-site
USD 165,000 - 224,000
Health insurance
401(k) matching
Paid time off
+2
Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Annapurna Labs • San Diego (CA)

On-site
USD 144,000 - 194,000
Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Annapurna Labs • Austin (TX)

On-site
USD 144,000 - 194,000
Sr. Software Development Engineer, Trainium Systems
Sr. Software Development Engineer, Trainium Systems

Annapurna Labs • Cupertino (CA)

On-site
USD 193,000 - 262,000
Health insurance
RSUs (Restricted Stock Units)
401(k) matching
Software Development Engineer - Embedded Systems
Software Development Engineer - Embedded Systems

Amazon Inc. • Seattle (WA)

On-site
USD 144,000 - 194,000
Health insurance
401(k) matching
Paid time off
+1
Software Development Engineer, Trainium Systems
Software Development Engineer, Trainium Systems

Amazon.com Inc. • Seattle (WA)

On-site
USD 165,000 - 224,000
Software Development Engineer, Annapurna Labs, Elastic Collectives
Software Development Engineer, Annapurna Labs, Elastic Collectives

Annapurna Labs • Cupertino (CA)

On-site
USD 150,000 - 250,000
Systems Development Engineer (AWS Generative AI & ML Servers), AWS HW Engineering
Systems Development Engineer (AWS Generative AI & ML Servers), AWS HW Engineering

Amazon Web Services (AWS) • Seattle (WA)

On-site
USD 129,000 - 175,000