Distributed Systems Engineer for AI Inference Platform

Baseten; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Baseten is looking for distributed systems engineers and generalists to build the distributed runtime powering large-scale LLM inference on Baseten. You’ll enable customers to deploy models with industry-leading performance, scalability, and reliability via Model APIs and hosted endpoints.

You’ll work across the stack—from developer experience used to deploy models to the systems that orchestrate deployments on Kubernetes and route traffic efficiently.

Responsibilities

• Build infra and orchestration for large-scale distributed LLM inference (routing, autoscaling, scheduling, runtime management).
• Design, build Model APIs with advanced inference: structured outputs, tool calling, and multimodal serving.
• Implement API versioning, validation, usage metering, quotas, and authentication.
• Instrument observability and benchmarks for speed, reliability, and quality.
• Define testing, release automation, and operational practices.
• Debug and harden production systems across Kubernetes, runtimes, networking, and GPU workloads.
• Collaborate with teams to roll out optimizations to customers.
• Own projects end to end from architecture to deployment and monitoring.

Job description

Baseten is looking for distributed systems engineers and generalists to build the distributed runtime powering large-scale LLM inference on Baseten. You’ll enable customers to deploy models with industry-leading performance, scalability, and reliability via Model APIs and hosted endpoints.

You’ll work across the stack—from developer experience used to deploy models to the systems that orchestrate deployments on Kubernetes and route traffic efficiently.