Distributed Systems Engineer for LLM Inference Platform

Baseten; San Francisco (CA); On-site; USD 180,000 - 360,000; Full time

Benefits offered by this job

Competitive equity

Medical insurance

Flexible PTO

Parental leave

Fertility stipend

401(k)

Learning opportunities

Job summary

Baseten, a leader in distributed AI infrastructure, seeks engineers to build the runtime powering large-scale LLM inference and hosted endpoints. You’ll impact model deployment, tooling, and performance across the platform.

You’ll own end-to-end projects from architecture to deployment, balancing performance with reliability and developer experience, while collaborating across teams on challenging integration tasks.

Qualifications

• Bachelor’s, Master’s, or Ph.D. in Computer Science, Engineering, or related field, or equivalent practical experience.
• 3+ years building and operating distributed systems, backend infrastructure, or large-scale APIs.

Responsibilities

• Build infrastructure and orchestration systems for large-scale distributed LLM inference, including routing, autoscaling, scheduling, and runtime management.
• Design, build, and operate Model APIs with advanced inference capabilities: structured outputs, tool/function calling, multimodal serving.
• Implement platform fundamentals such as API versioning, validation, usage metering, quotas, and authentication.

Skills

Distributed systems

Backend services

Low-latency design

Dev experience focus

Collaboration

Education

Bachelor's/Master/PhD in CS/Engineering

Equivalent practical experience

Tools

Kubernetes

APIs & gateways

CI/CD

Job description

Baseten, a leader in distributed AI infrastructure, seeks engineers to build the runtime powering large-scale LLM inference and hosted endpoints. You’ll impact model deployment, tooling, and performance across the platform.

You’ll own end-to-end projects from architecture to deployment, balancing performance with reliability and developer experience, while collaborating across teams on challenging integration tasks.