AI Inference Systems Engineer

Engg; San Francisco (CA); On-site; USD 150,000 - 190,000; Full time

Benefits offered by this job

Unlimited PTO

Medical, dental, and vision benefits

Life insurance and disabili

Job summary

Sierra’s AI agents rely on foundation models; as a Software Engineer on Inference, you’ll help define the inference architecture across self-hosted models and third‑party providers. You’ll work on systems for serving and routing inference, managing capacity and quota, and optimizing latency, reliability, and cost.

This is a systems‑first role at the intersection of distributed infrastructure and AI. You don’t need to be an ML researcher—but you should love solving complex problems and applying

Qualifications

• Deep systems thinking and strong distributed systems fundamentals.
• Experience designing, building, and operating large‑scale production systems.
• Strong judgment around tradeoffs involving latency, reliability, capacity, and cost.
• Experience taking ownership of complex infrastructure from architecture through production operation.
• Excitement about applying systems expertise to AI infrastructure and learning quickly as the underlying technology evolves.

Responsibilities

• Partner with frontier labs and providers. At our scale, we rely on frontier labs, and inference providers to supply capacity, training and inference infrastructure.
• Shape Sierra’s inference architecture. Design how inference traffic flows across models, infrastructure, and providers, including new serving and proxy layers as Sierra scales.
• Build for low latency and high reliability. Develop systems for routing, failover, capacity management, and quota that keep inference performant and available across large‑scale production workloads.
• Build and operate self-hosted inference. Run models on GPU infrastructure, from building containers and operating inference engines to managing the underlying compute capacity.
• Optimize inference performance. Work with the Applied Research team on techniques such as speculative decoding and serving‑engine optimizations that improve latency, throughput, and cost.
• Build across a hybrid inference stack. Work with both Sierra‑managed infrastructure and leading inference platforms, making architectural decisions about where and how workloads should run.
• Push the serving stack forward. Work closely with inference providers to tune engines and infrastructure for Sierra’s workloads.
• Support the broader model lifecycle. Contribute to infrastructure that enables post‑training while partnering closely with our Models and Agent Runtime teams.

Skills

Distributed systems

Self-hosted inference

Latency optimization

GPU infrastructure

Job description

Sierra’s AI agents rely on foundation models; as a Software Engineer on Inference, you’ll help define the inference architecture across self-hosted models and third‑party providers. You’ll work on systems for serving and routing inference, managing capacity and quota, and optimizing latency, reliability, and cost.

This is a systems‑first role at the intersection of distributed infrastructure and AI. You don’t need to be an ML researcher—but you should love solving complex problems and applying