Software Development Engineer, ML Acceleration, Trainium AI Systems, Annapurna Labs

Annapurna Labs (U.S.) Inc.

Austin (TX)

On-site

USD 120,000 - 190,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Annapurna Labs is hiring a Software Development Engineer II to build data and analytics systems for diagnostic tests across our server fleet. You will ingest telemetry and repair data, create dashboards, and develop metrics and alarms to surface test behavior and failure patterns at scale.

You will work with hardware, firmware, provisioning, and data center operations teams, writing production code in Python and SQL, and shaping the evidence that gates production decisions.

Qualifications

  • 3+ years of non-internship professional software development experience.
  • 2+ years of design or architecture experience for new and existing systems.
  • Bachelor’s degree or foreign equivalent in CS, Engineering, Mathematics, or a related field.
  • Experience programming with at least one software programming language.

Responsibilities

  • Design, build, and operate production data pipelines ingesting diagnostic, telemetry, and repair-ticket data.
  • Build and own the metrics, alarms, and anomaly detection that surface test regressions, failure-rate shifts, and new failure signatures across hardware generations without a human looking for them.
  • Build dashboards and visualizations that make fleet and test health legible to engineers, hardware partners, and leadership, covering failure rates, failure-signature breakdowns, first pass yield, and repair latency.
  • Analyze large-scale fleet data to find root cause behind failure trends, and separate genuine hardware faults from software defects and test noise.
  • Define the evidence standard that gates operational decisions, including whether a diagnostic has soaked long enough and cleanly enough in the fleet to move from observation mode into blocking production.
  • Improve data quality and pipeline reliability so downstream consumers trust the numbers without re-deriving them.
  • Write clear analyses and design documents for technical and non-technical readers, including leadership.

Skills

Software development
System design/architecture
Programming language

Education

Bachelor’s degree (CS/Engineering/Math)

Job description

Annapurna Labs designs the silicon behind AWS machine learning acceleration. Trainium and Inferentia servers train and serve the largest models our customers run, and before any of that hardware carries customer traffic it has to prove it works. MLA Vetting owns that gate. We build the diagnostic tests that every accelerator server runs before it becomes sellable, and we decide which failures block a host, which route to a technician for repair, and which are noise.

That gate only works if we can see through it. We are hiring a Software Development Engineer II to build the data and analytics systems that tell us what our tests are actually doing across the fleet. You will build the pipelines that ingest diagnostic results from tens of thousands of servers, the dashboards that make test behavior legible, and the metrics and alarms that tell us a test has started failing on a hardware generation before it costs us a week of capacity. When the team decides whether a diagnostic has enough evidence to move from observation into blocking production, your data answers that question.

This is a software engineering role, not a reporting role. You will write and operate production code, own the alarms and metrics that page us, and build detection for trends nobody is watching yet. You will work directly with hardware, firmware, provisioning, and data center operations teams who act on what you build.

You are fluent in Python and SQL and have built production data pipelines at scale. Experience with Amazon Redshift, Apache Spark, workflow orchestration, and dashboarding in Grafana or QuickSight maps directly to what you will work on here. Experience with telemetry or diagnostic data from a large server fleet, or with defining metrics and anomaly detection on operational time-series data, will have you productive faster. Familiarity with server hardware or data center operations is a plus, not a requirement - we will teach you the hardware.

Key job responsibilities
  • Design, build, and operate production data pipelines that ingest diagnostic, telemetry, and repair-ticket data from the Trainium and Inferentia fleet into a warehouse other teams query with confidence.
  • Build and own the metrics, alarms, and anomaly detection that surface test regressions, failure-rate shifts, and new failure signatures across hardware generations without a human going looking for them.
  • Build dashboards and visualizations that make fleet and test health legible to engineers, hardware partners, and leadership, covering failure rates, failure-signature breakdowns, first pass yield, and repair latency.
  • Analyze large-scale fleet data to find root cause behind failure trends, and separate genuine hardware faults from software defects and test noise - a distinction that decides whether a failure reaches a technician or an engineer.
  • Define the evidence standard that gates operational decisions, including whether a diagnostic has soaked long enough and cleanly enough in the fleet to move from observation mode into blocking production.
  • Improve data quality and pipeline reliability so downstream consumers trust the numbers without re-deriving them.
  • Write clear analyses and design documents for technical and non-technical readers, including leadership.
A day in the life

You start by checking an alarm that fired overnight: a diagnostic's failure rate climbed on one server generation. You query the failure signatures, find the increase concentrates in one component and one data center, and hand the breakdown to the hardware team by mid-morning. The rest of the day goes to a pipeline you are building to join repair outcomes against diagnostic results, so the team can measure how often a repair actually fixes the fault it was dispatched for. You close the day reviewing a teammate's code review and answering an operations partner who needs a new cut of yield data.

About the team

MLA Vetting sits between hardware manufacturing and customer-ready capacity. We own the diagnostic test suites and the triage framework that turn a raw hardware failure into a specific, actionable repair, and we own the fleet-scale evidence that drives hardware and firmware improvements upstream. Our work is measured in how fast a server reaches sellable and how often it gets there on the first try.

Basic Qualifications
  • 3+ years of non-internship professional software development experience
  • 2+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience
  • Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field
  • Experience programming with at least one software programming language
Preferred Qualifications
  • 3+ years of full software development life cycle, including coding standards, code reviews, source control management, build processes, testing, and operations experience
  • Bachelor's degree in computer science or equivalent

Amazon is an equ

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Development Engineer II, Data Engineering, Annapurna Labs
Software Development Engineer II, Data Engineering, Annapurna Labs

Amazon Inc. • Austin (TX)

On-site
USD 144,000 - 194,000
Software Development Engineer II, Data Engineering, Annapurna Labs
Software Development Engineer II, Data Engineering, Annapurna Labs

Annapurna Labs • Austin (TX)

On-site
USD 144,000 - 194,000
Health insurance
RSUs
Paid time off
+1
AI Hardware Systems Engineer, Annapurna Labs, Trainium Machine Learning Fleet Operations
AI Hardware Systems Engineer, Annapurna Labs, Trainium Machine Learning Fleet Operations

Annapurna Labs (U.S.) Inc. • Austin (TX)

On-site
USD 150,000 - 190,000
AI Hardware Systems Engineer, Annapurna Labs, Trainium Machine Learning Fleet Operations
AI Hardware Systems Engineer, Annapurna Labs, Trainium Machine Learning Fleet Operations

Amazon Web Services (AWS) • Austin (TX)

On-site
USD 136,000 - 184,000
Health insurance
RSUs / restricted stock units
401(k) match
+2
Sr. System Development Engineer, High-Performance Accelerator Servers for AI/ML
Sr. System Development Engineer, High-Performance Accelerator Servers for AI/ML

Amazon Data Services, Inc. • Austin (TX)

On-site
USD 180,000 - 240,000
Cloud Hardware Development Engineer - Manufacturing Test Engineering, Annapurna Labs, Manufacturing Test Engineering
Cloud Hardware Development Engineer - Manufacturing Test Engineering, Annapurna Labs, Manufacturing Test Engineering

Annapurna Labs (U.S.) Inc. • Austin (TX)

On-site
USD 110,000 - 160,000
Senior Embedded Software Engineer, Machine Learning Accelerator Technology
Senior Embedded Software Engineer, Machine Learning Accelerator Technology

Annapurna Labs (U.S.) Inc. • Austin (TX)

On-site
USD 140,000 - 180,000
Software Development Engineer II - Regression Infrastructure, Trainium Manufacturing Infrastucture
Software Development Engineer II - Regression Infrastructure, Trainium Manufacturing Infrastucture

Annapurna Labs (U.S.) Inc. • Austin (TX)

On-site
USD 150,000 - 210,000
Lead Engineer for Manufacturing and Datacenter Lab, Trainium Manufacturing, Quality and Reliability
Lead Engineer for Manufacturing and Datacenter Lab, Trainium Manufacturing, Quality and Reliability

Annapurna Labs (U.S.) Inc. • Austin (TX)

On-site
USD 140,000 - 180,000
Software Development Engineer, Annapurna Labs, Elastic Collectives
Software Development Engineer, Annapurna Labs, Elastic Collectives

Annapurna Labs (U.S.) Inc. • Cupertino (CA)

On-site
USD 180,000 - 240,000