Software Engineer, Inference Stack (LLM Infra)

Baseten; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Baseten is seeking a Software Engineer for the Inference Stack to own distributed infrastructure for large-scale LLM inference. You will navigate the stack from developer-facing features to low-level systems, enabling robust, scalable model deployments.

You’ll work on routing, autoscaling, observability, and runtime management while collaborating with Model Performance engineers to push optimizations broadly to customers. This role emphasizes reliability, performance, and developer experience.

Qualifications

• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field.
• Strong background in distributed systems, backend infrastructure, or platform engineering.
• Proven experience building production-grade systems.

Responsibilities

• Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference.
• Work across the stack from customer-facing features to low-level infrastructure components.
• Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management.
• Improve the reliability, scalability, and usability of the inference stack.
• Collaborate with Model Performance engineers to make inference optimizations broadly available to customers and easy to configure.
• Help define best practices around testing, release automation, benchmarking, and operational excellence.
• Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads.
• Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience.
• Own projects end-to-end: from architecture and implementation through deployment, monitoring, and iteration based on customer feedback

Skills

Distributed systems

Backend infrastructure

Platform engineering

Production experience

Education

Bachelor's/Master's/PhD in CS/Engineering or related

Advanced degree preferred

Tools

Kubernetes

Distributed runtimes

Job description

Baseten is seeking a Software Engineer for the Inference Stack to own distributed infrastructure for large-scale LLM inference. You will navigate the stack from developer-facing features to low-level systems, enabling robust, scalable model deployments.

You’ll work on routing, autoscaling, observability, and runtime management while collaborating with Model Performance engineers to push optimizations broadly to customers. This role emphasizes reliability, performance, and developer experience.