Runtime Engineer

matx

United States

Hybrid

USD 250,000 - 475,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

4 weeks PTO (accrued)
Health insurance
401K with company match
Professional Development Budget
Remote work eligibility (up to 3 weeks

Job summary

MatX is building custom silicon for LLM inference and training, delivering a host-side interface library and a robust runtime contract stack. You will design executables, ABIs, and kernel interfaces while enabling Python tensor flow with PyO3 bindings and C-ABI bridges.

The role involves building a scalable LLM serving stack, optimizing host-device interactions, and ensuring performance targets across devices and racks. Strong systems programming and ML-systems literacy are required.

Qualifications

  • Must have systems programming experience with memory management and allocator design.
  • Experience building Python interop layers for production use.
  • Experience designing and maintaining API/ABI contracts between teams.
  • Hands-on with at least one accelerator model (CUDA, ROCm, oneAPI, TPU).
  • Familiarity with ML training and inference loops and tensor layouts.

Responsibilities

  • Build host-side interface library: memory mgmt, DMA, streams/events, sync primitives.
  • Own and extend executable formats and compiler→runtime contracts.
  • Design custom-kernel ABI and host-side marshaling (DLPack, numpy) to move Python tensors to device.
  • Create Python bindings via PyO3 with a C-ABI shim.
  • Build LLM inference serving stack and cluster orchestration primitives.
  • Ensure robust host interconnect topology and failure-detection across racks.
  • Define chip exposure to profilers and debuggers and hit performance targets.

Skills

Systems programming
Python interop
FFI/ABI
CUDA / accelerators
API/ABI contracts

Job description

What MatX is Building

MatX is building custom silicon for large-language-model inference and training, with HW/SW co-design across ISA, RTL, simulator, compiler, and kernels so each layer benefits from the others. The runtime owns the host-side stack and the contracts that bind those teams together.

What You'll Do Here

Build the host-side interface library - device memory management, DMA, streams and events, sync primitives - that every compiler-emitted program runs on top of

Own and extend the executable format: the compiler→runtime contract, its versioning, the weight and quantization layouts that let compiler and runtime evolve independently

Design the custom-kernel ABI - calling convention, sync semantics, lifecycle - and the host-side marshaling layer (DLPack, the buffer protocol, numpy) that gets Python tensors to the device

Build Python bindings via PyO3, with a C-ABI shim as the alternative integration path for downstream consumers

Build the LLM inference serving stack - paged KV cache, continuous batching, request scheduling, token streaming - and the cluster orchestration primitives underneath it

Bring up interconnect topology from the host and own the failure-detection and clean-teardown path for stop-restructure-resume recovery across racks

Design what the chip exposes to host-side profilers and debuggers - perf counters, traces, and the Python surfaces ML engineers actually use - and hit measurable performance targets on runtime overhead and serving throughput

Who You Are
  • Strong experience in a systems programming language - Rust, C, C++, or Go - including memory management, allocator design, and FFI/ABI work
  • Have built Python interop layers in production (PyO3, ctypes, pybind11, or equivalent C-ABI bridging)
  • Have designed and maintained API or ABI contracts between teams - versioning, evolution, breaking-change discipline - not just consumed someone else's
  • Hands-on with at least one accelerator programming model (CUDA, ROCm, oneAPI Level Zero, TPU, or comparable) - enough to reason about device memory, async execution, and kernel launch
  • ML-systems literate - comfortable with the training and inference loop, what collectives do, what a tensor layout is. Research depth not required.
Bonus Points If You Have
  • LLM inference internals - vLLM, TensorRT-LLM, or SGLang (paged attention, scheduler design)
  • Rust at depth, including proc macros, unsafe with soundness reasoning, and complex lifetime/trait work
  • Custom allocator design (slab, paged, arena) or other low-level memory work
  • ML framework integration experience (PyTorch custom backends, JAX/XLA, ONNX runtime)
  • Profiler or tracing infrastructure work (perfetto, Nsight, or a custom stack)
  • Driver-adjacent or kernel-bypass work, or prior new-silicon bring-up
Compensation
  • Early Career - $160,000 - $250,000 + equity
  • Mid Career - $175,000 - $362,500 + equity
  • Senior Career - $250,000 - $475,000 + equity
What We Offer
  • Time off: 4 weeks PTO (accrued) + 12 company Holidays + up to 3 weeks remote work
  • Health: Company-subsidized Medical (Kaiser or Anthem) for employees & dependents, Guardian Dental and Vision insurances for employee & dependents, and life insurance (employee only), plus HSA and FSA offerings via Lively.
  • Financial Security: Choose from Roth or pre-tax 401K (or both) with a 5% company contribution (even if you don't contribute), 100% company-paid life insurance (up to $300K), long-term disability and travel insurance.
  • Professional Development: $1500 per year Professional Development Budget
  • Team Meals: MatX provides onsite team lunch & dinner Monday - Friday, with your choice of ordering via WeBox, Specialty's or via our reimbursement system
  • Commute on Us: We pay 100% for your daily commute. Commute via our company rideshare account, reimburse public transportation, or get a Fasttrak card (or all three)
  • MatX E[x]tras: $50/mo to use on the perk you value most
  • Cell & Internet Reimbursement: $35/mo for cellular and $40/mo for wifi
  • Remote Home Setup: Use our Amazon business account for up to $1000 in office setup purchases.
  • Mental Wellbeing: 100% paid mental health benefit via SpringHealth and Guardian EAP.
  • Support to Parents: Up to 12 weeks paid parental leave regardless of path to parenthood, 10 weeks pregnancy disability leave, flexible return-to-work hours, and a $6K Family Fund to cover fertility, surrogacy, adoption, menopausal health, preservation, family counseling and more.

As part of our dedication to the diversity of our team and our focus on creating an inviting and inclusive work experience, MatX is committed to a policy of Equal Employment Opportunity and will not discriminate against an applicant or employee on the basis of race, color, religion, creed, national origin or

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Infrastructure Engineer
Infrastructure Engineer

MatX • Mountain View (CA)

Hybrid
USD 160,000 - 600,000
PTO (4 weeks)
Holidays (12)
Remote work up to 3 weeks
+2
Senior SoC Power Analysis & Optimization Engineer
Senior SoC Power Analysis & Optimization Engineer

matx • United States

On-site
USD 160,000 - 600,000
4 weeks PTO
12 company holidays
Remote work up to 3 weeks
+2
Infrastructure Engineer, AI Tooling
Infrastructure Engineer, AI Tooling

matx • United States

On-site
USD 160,000 - 600,000
4 weeks PTO
Company Holidays
Remote work up to 3 weeks
+10
Hardware Tools Engineer
Hardware Tools Engineer

MatX • United States

Remote
USD 250,000 - 600,000
4 weeks PTO
Company health insurance
401(k) with company match
+4
MatX Internships 2027
MatX Internships 2027

MatX • Mountain View (CA)

On-site
USD 136,000 - 184,000
4 weeks PTO
12 holidays
Remote work up to 3 weeks
+12
Software Engineer - Kernels
Software Engineer - Kernels

matx • United States

Hybrid
USD 160,000 - 600,000
PTO & Holidays + remote work up to 3 w
Medical insurance for employees and +2
5% company 401K contribution
System Software Engineer, Linux Driver and Driver Adjacent
System Software Engineer, Linux Driver and Driver Adjacent

MatX • Mountain View (CA)

On-site
USD 160,000 - 275,000
4 weeks PTO
12 company holidays
Remote work up to 3 weeks
+7
SOC Micro-Architect and RTL Designer
SOC Micro-Architect and RTL Designer

MatX Inc. • Mountain View (CA)

Hybrid
USD 160,000 - 600,000
PTO & Holidays
Medical insurance
Dental & Vision
+3
Microarchitect and RTL Designer - Compute
Microarchitect and RTL Designer - Compute

MatX • Mountain View (CA)

Hybrid
USD 175,000 - 600,000
4 weeks PTO (accrued)
Remote work up to 3 weeks
Medical, dental, vision insurance
Hardware Tools Engineer
Hardware Tools Engineer

MatX • Mountain View (CA)

Hybrid
USD 160,000 - 600,000
4 weeks PTO
HSA/Medical Insurance
401K with company contribution
+2