Distributed Systems Engineer, Multi-Chip Simulation (Mid/Senior/Principal)

Positron AI

United States

On-site

USD 160,000 - 350,000

Full time

4 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Health and wellbeing benefits
Company-paid insurance (medical, life,
Disability coverage

Job summary

Positron AI in the United States is seeking a Senior Distributed Systems Engineer to design and scale a multi-chip simulation stack before the hardware exists. You will own the simulators that model host-to-chip and chip-to-chip interactions, balancing fidelity with runtime for daily CI workflows.

You will work with the chip modeling, SDK, and runtime teams to model timing, back-pressure, and partial failures, and you will correlate results with emulation and hardware measurements.

Qualifications

  • 5+ years of experience as a distributed systems engineer building and operating software that coordinates many machines or accelerators through RPC, message passing, consensus, or collective communication.
  • Strong understanding of the failure modes of distributed software, including races, partial failure, ordering and consistency, timeouts and retries, back-pressure, and deadlock.
  • Demonstrated experience debugging timing-dependent bugs across hosts, NICs, or accelerators using traces and logs.
  • Experience building simulators, emulators, or test harnesses for distributed or networked systems is strongly preferred.
  • Sound judgment in choosing the right level of fidelity for a question and balancing accuracy against run time.
  • Strong Rust skills, or expert-level C/C++ skills and a willingness to work in Rust day to day, along with proficiency in Python for tooling and analysis.
  • Excellent communication skills, with the ability to explain what the simulation shows, what it does not, and what the software or hardware team should change.

Responsibilities

  • Design and build the simulators for the host-to-chip and chip-to-chip layers of our system, including the drivers, links, protocols, and transfers that connect many chips into one machine.
  • Model the communication fabric, including transfer timing, collectives, ordering guarantees, back-pressure, congestion, and partial failure, calibrated against real hardware.
  • Own the architecture of the multi-chip simulation stack as new chips, interconnects, and deployment topologies arrive.
  • Compose simulators of different fidelity (fast functional, cycle-level, and hardware emulation) into one coherent multi-chip simulation.
  • Scale the simulation from a few chips to full racks while keeping runs fast enough for daily use in CI.
  • Correlate simulated behavior against emulation and hardware measurements, with regression coverage and clear documentation of assumptions and limitations.

Skills

Distributed systems
RPC & messaging
Rust
C/C++
Python
Performance debugging
Technical communication

Tools

Cadence Palladium
Synopsys ZeBu
FPGA prototyping

Job description

Role Overview

Running one AI model across many chips is a distributed systems problem: a host and many chips exchanging data over links, waiting on each other, and failing in ways no single chip would. As a Senior Distributed Systems Engineer, you will help us simulate that system before the hardware exists. You will work with the chip modeling, SDK, and runtime teams to build the simulators that stand in for the host, the links, and the protocols between chips.

This role is about building and scaling simulation infrastructure, not testing someone else's code. You will own the software architecture of the multi-chip simulation stack, decide how much fidelity each layer needs, and keep it fast enough that distributed software is exercised every day in CI.

Key Responsibilities

Multi-Chip Simulator Design

Design and build the simulators for the host-to-chip and chip-to-chip layers of our system, including the drivers, links, protocols, and transfers that connect many chips into one machine.

Model the communication fabric, including transfer timing, collectives, ordering guarantees, back-pressure, congestion, and partial failure, calibrated against real hardware.

Architecture and Scale

Own the architecture of the multi-chip simulation stack as new chips, interconnects, and deployment topologies arrive.

Compose simulators of different fidelity (fast functional, cycle-level, and hardware emulation) into one coherent multi-chip simulation.

Scale the simulation from a few chips to full racks while keeping runs fast enough for daily use in CI.

Correlation and Quality

Correlate simulated behavior against emulation and hardware measurements, with regression coverage and clear documentation of assumptions and limitations.

Required Qualifications
  • 5+ years of experience as a distributed systems engineer building and operating software that coordinates many machines or accelerators through RPC, message passing, consensus, or collective communication.
  • A strong understanding of the failure modes of distributed software, including races, partial failure, ordering and consistency, timeouts and retries, back-pressure, and deadlock.
  • Demonstrated experience debugging timing-dependent bugs across hosts, NICs, or accelerators using traces and logs.
  • Experience building simulators, emulators, or test harnesses for distributed or networked systems is strongly preferred.
  • Sound judgment in choosing the right level of fidelity for a question and balancing accuracy against run time.
  • Strong Rust skills, or expert-level C/C++ skills and a willingness to work in Rust day to day, along with proficiency in Python for tooling and analysis.
  • Excellent communication skills, with the ability to explain what the simulation shows, what it does not, and what the software or hardware team should change.
Preferred Qualifications
  • Hands-on experience with multi-GPU or multi-accelerator inference or training, including NCCL-style collectives, tensor, pipeline, and expert parallelism, and host-device runtimes.
  • Familiarity with hardware emulators such as Cadence Palladium or Synopsys ZeBu, or with FPGA prototyping.
Leveling & Scope

While this role is currently posted at a specific level, we are a growth-oriented organization and are open to hiring at a more senior level for the right candidate. Please note that this job description serves as a focused but generalized overview of the role; specific responsibilities and impact expectations will be tailored to the experience and seniority of the final hire.

Why Join Us?

You will own the architecture of the simulation stack that lets our distributed software run across racks of chips long before the hardware exists.

You will work across chip modeling, SDK, and runtime teams on one of the hardest problems in AI inference: making many chips behave as one machine.

Compensation & Benefits

The base salary range for this role is Mid: $160,000-$220,000, Senior: $220,000-$270,000, Principal: $270,000-$350,000.

Please note that the figures provided represent the base salary range only and do not include other elements of our total compensation package, equity, or comprehensive benefits.

At Positron AI, we value the unique expertise each candidate brings. While the range above reflects our typical expectation for the position, we reserve the flexibility to exceed this range for candidates whose specialized skills, significant experience, or unique qualifications fall outside the standard scope of the role. Final offers are determined based on a variety of factors, including internal equity, and individual impact.

Benefits & Perks

We want you to do your best work and feel confident that you and your family are taken care of. That means comprehensive coverage, real time to rest, and support for your future.

Health and wellness

Fully company-paid medical, dental, and vision insurance for you and your dependents.

Company-paid life and disability coverage, with voluntary options to add more.

Supplemental hospital, critical il

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Modeling & Simulation Engineer
Modeling & Simulation Engineer

Positron • United States

Remote
USD 200,000 - 300,000
Unlimited PTO
Remote first culture
401(k) with company matching
+1
ML Systems Engineer, Model Architectures (Mid/Senior/Principal)
ML Systems Engineer, Model Architectures (Mid/Senior/Principal)

Positron AI • United States

On-site
USD 160,000 - 350,000
Fully company-paid medical, dental, &
Company-paid life/disability
Unlimited PTO
+2
ASIC Architect, Principal (AI Inference)
ASIC Architect, Principal (AI Inference)

Positron AI • United States

Remote
USD 300,000 - 350,000
Health insurance
Dental insurance
Vision insurance
+3
Physical Design Engineer - Synth, Senior
Physical Design Engineer - Synth, Senior

Positron AI • United States

Hybrid
USD 200,000 - 300,000
Health insurance
Unlimited paid time off
Remote-first culture
+1
Distributed Systems Engineer, Multi-Chip Simulation (Mid/Senior/Principal)
Distributed Systems Engineer, Multi-Chip Simulation (Mid/Senior/Principal)

positron.ai • United States

Remote
USD 140,000 - 210,000
Remote-first
Company-provided computer
Medical, dental, and vision coverage
+2
AI Forward Deployed Engineer, Lead
AI Forward Deployed Engineer, Lead

Positron AI • United States

Remote
USD 325,000 - 400,000
Medical, dental, and vision insurance
Life and disability insurance
Supplemental coverage options
+4
ASIC Physical Design Engineer - Power, Senior
ASIC Physical Design Engineer - Power, Senior

Positron • United States

Remote
USD 200,000 - 300,000
Health and wellness
Time off and flexibility
Compensation and future
+2
Software Engineer, Control Plane (Senior/Principal)
Software Engineer, Control Plane (Senior/Principal)

Positron AI • United States

Remote
USD 170,000 - 325,000
Health insurance
401(k) with company matching
On-site lab setup
Systems Engineer, Experiments & Integration (Mid/Senior/Principal)
Systems Engineer, Experiments & Integration (Mid/Senior/Principal)

Positron • United States

Remote
USD 160,000 - 350,000
Health insurance
Life & disability
Remote-first
+3
Software Engineer, Control Plane (Senior/Principal)
Software Engineer, Control Plane (Senior/Principal)

Positron Corporation • Spokane Valley (WA)

On-site
USD 170,000 - 325,000
Health insurance
401(k) matching
Unlimited PTO
+1