Senior Distributed Systems Engineer — AI Inference Platform

Baseten; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity

Comprehensive medical for US

Flexible PTO

Parental leave

Fertility stipend

401(k)

Learning opportunities

Job summary

Baseten powers mission-critical AI inference at scale. We seek distributed systems engineers to own the runtimes and APIs that run large-scale LLM inference and hosted endpoints.

You’ll work across the stack from developer tooling to orchestration on Kubernetes, driving speed, reliability, and cost-efficiency for customers. You’ll collaborate with the team on model serving, inference performance, and API design while shipping robust, production-grade infrastructure and tools for a rapidly

Qualifications

• 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.
• “Own low-latency, reliable backend services including rate limiting, auth, quotas, metering, migrations.”
• Experience with performance profiling, tracing, capacity planning, and SLO management.
• Strong written communication and collaboration across teams.
• Interest in inference engineering; ML or model serving experience is a plus.

Responsibilities

• Build infrastructure and orchestration systems for large-scale distributed LLM inference.
• Design, build, and operate Model APIs with JSON outputs and tool calling.
• Implement platform fundamentals: API versioning, validation, metering, quotas, authentication.
• Instrument observability and run benchmarks for speed, reliability, and quality.
• Debug and harden production systems spanning Kubernetes, runtimes, networking, and GPUs.
• Collaborate with Inference Performance engineers to broaden optimization reach.
• Own projects end to end from architecture to deployment and iteration with customers.

Skills

Distributed systems

Backend infrastructure

Low latency

Developer experience

Code ownership

Observability

Collaboration

Performance tuning

Education

Bachelor's/Master's/Ph.D. in CS or related

Tools

Kubernetes

API gateways

Observability tooling

CI/CD systems

Job description

Baseten powers mission-critical AI inference at scale. We seek distributed systems engineers to own the runtimes and APIs that run large-scale LLM inference and hosted endpoints.

You’ll work across the stack from developer tooling to orchestration on Kubernetes, driving speed, reliability, and cost-efficiency for customers. You’ll collaborate with the team on model serving, inference performance, and API design while shipping robust, production-grade infrastructure and tools for a rapidly