Software Engineer — Inference Stack for Scalable LLMs

BaseTen; New York, San Francisco (NY, CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity

Health insurance

Flexible PTO

Parental leave

401(k)

Job summary

Baseten in New York is seeking a Software Engineer for the Inference Stack to advance large-scale LLM inference. You’ll work across the stack—from developer-facing features to low-level infrastructure, Kubernetes deployments, and traffic routing.

You’ll own production systems and solve hard integration challenges to keep our platform fast, reliable, and easy to use. You should be excited by building in production, collaborating with Model Performance engineers, and shaping testing, release

Qualifications

• Experience building and operating production systems with high reliability, latency, and scale.
• Strong focus on developer experience and how end users interact with systems.
• Willingness to learn new languages, frameworks, and systems as needed.
• Ability to debug complex multi-layered systems across stacks.
• Excellent collaboration and communication skills.

Responsibilities

• Develop infrastructure and orchestration systems for deploying large-scale distributed LLM inference.
• Work across the stack—from customer-facing features to low-level infrastructure components.
• Build platform capabilities for routing, autoscaling, scheduling, observability, and runtime management.
• Improve reliability, scalability, and usability of the inference stack.
• Collaborate with Model Performance engineers to deploy inference optimizations for customers.
• Define testing, release automation, benchmarking, and operational practices.
• Debug production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads.
• Make thoughtful engineering tradeoffs balancing performance, reliability, and developer experience.
• Own projects end-to-end: architecture, implementation, deployment, monitoring, iteration.

Skills

Distributed systems

Backend infrastructure

Platform engineering

Developer experience

Debugging complex systems

Communication skills

Education

Bachelor’s/Master’s/PhD in CS or related

Job description

Baseten in New York is seeking a Software Engineer for the Inference Stack to advance large-scale LLM inference. You’ll work across the stack—from developer-facing features to low-level infrastructure, Kubernetes deployments, and traffic routing.

You’ll own production systems and solve hard integration challenges to keep our platform fast, reliable, and easy to use. You should be excited by building in production, collaborating with Model Performance engineers, and shaping testing, release