LLM Inference Platform Engineer

Baseten; New York (NY); On-site; USD 216,000 - 360,000; Full time

Benefits offered by this job

Equity

Comprehensive health coverage

Flexible PTO

Parental leave

Fertility stipend

401(k)

Learning opportunities

Job summary

Baseten in the United States is hiring a Software Engineer for the Inference Stack to build distributed runtime for large‑scale LLM inference. You’ll own production systems spanning Kubernetes, routing, autoscaling, observability, and deployment tooling, with a focus on reliability and developer experience.

You’ll collaborate across teams to push performance improvements, scale, and user‑facing ease‑of‑use features, joining a fast‑moving platform engineering group at the forefront of AI

Qualifications

• Strong background in distributed systems, backend infrastructure, or platform engineering.
• Experience building and operating production systems where reliability, latency, and scale are first‑class concerns.
• Excellent communication and collaboration skills.

Responsibilities

• Develop infrastructure and orchestration systems for deploying and managing large‑scale distributed LLM inference
• Work across the stack, from customer‑facing features to low‑level infrastructure components
• Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management
• Improve the reliability, scalability, and usability of our inference stack
• Collaborate with Model Performance engineers to make inference optimizations broadly available
• Define best practices around testing, release automation, benchmarking, and operational excellence
• Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
• Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience
• Own projects end‑to‑end: architecture, implementation, deployment, monitoring, iteration based on customer feedback

Skills

Distributed systems

Backend infrastructure

Platform engineering

Developer experience

Debugging complex systems

Education

Bachelor's, Master's, or Ph.D. in CS/Engineering

Tools

Kubernetes

Job description

Baseten in the United States is hiring a Software Engineer for the Inference Stack to build distributed runtime for large‑scale LLM inference. You’ll own production systems spanning Kubernetes, routing, autoscaling, observability, and deployment tooling, with a focus on reliability and developer experience.

You’ll collaborate across teams to push performance improvements, scale, and user‑facing ease‑of‑use features, joining a fast‑moving platform engineering group at the forefront of AI