Software Engineer - Baseten Inference Stack

BaseTen; New York, San Francisco (NY, CA); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Equity

Health insurance

Flexible PTO

Parental leave

401(k)

Job summary

Baseten in New York is seeking a Software Engineer for the Inference Stack to advance large-scale LLM inference. You’ll work across the stack—from developer-facing features to low-level infrastructure, Kubernetes deployments, and traffic routing.

You’ll own production systems and solve hard integration challenges to keep our platform fast, reliable, and easy to use. You should be excited by building in production, collaborating with Model Performance engineers, and shaping testing, release

Qualifications

• Experience building and operating production systems with high reliability, latency, and scale.
• Strong focus on developer experience and how end users interact with systems.
• Willingness to learn new languages, frameworks, and systems as needed.
• Ability to debug complex multi-layered systems across stacks.
• Excellent collaboration and communication skills.

Responsibilities

• Develop infrastructure and orchestration systems for deploying large-scale distributed LLM inference.
• Work across the stack—from customer-facing features to low-level infrastructure components.
• Build platform capabilities for routing, autoscaling, scheduling, observability, and runtime management.
• Improve reliability, scalability, and usability of the inference stack.
• Collaborate with Model Performance engineers to deploy inference optimizations for customers.
• Define testing, release automation, benchmarking, and operational practices.
• Debug production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads.
• Make thoughtful engineering tradeoffs balancing performance, reliability, and developer experience.
• Own projects end-to-end: architecture, implementation, deployment, monitoring, iteration.

Skills

Distributed systems

Backend infrastructure

Platform engineering

Developer experience

Debugging complex systems

Communication skills

Education

Bachelor’s/Master’s/PhD in CS or related

Job description

ABOUT BASETEN

Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F, led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to to ship AI products.

THE ROLE

Baseten’s Inference Stack team builds the distributed runtime that powers large-scale LLM inference across our platform. We operate at the intersection of distributed systems, model performance, infrastructure, and developer experience. We enable customers to deploy and operate cutting-edge LLM models with industry-leading performance, scalability, reliability, and ease of use.

As a Software Engineer on the Inference Stack team, you’ll work across the stack—from the developer experience customers use to deploy models, the libraries used for features like tool calling and reasoning, all the way down to the systems we use to orchestrate deployments in Kubernetes and route traffic efficiently.

This is an ideal role for engineers who enjoy owning systems in production, solving hard integration problems, and making complex infrastructure simple and reliable for users.

EXAMPLE INITIATIVES

• https://www.baseten.co/blog/nvidia-dynamo-day-baseten-inference-stack/
• https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/
• https://www.baseten.co/blog/how-baseten-multi-cloud-capacity-management-mcm-powers-cloud-self-hosted-and-hybr/#comparing-deployment-options-cloud-vs-self-hosted-vs-hybrid

RESPONSIBILITIES

• Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference
• Work across the stack, from customer-facing features to low-level infrastructure components
• Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management
• Improve the reliability, scalability, and usability of our inference stack
• Collaborate closely with Model Performance engineers to make new inference optimizations broadly available to customers and easy to configure
• Help define best practices around testing, release automation, benchmarking, and operational excellence
• Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
• Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience
• Own projects end-to-end: from architecture and implementation through deployment, monitoring, and iteration based on customer feedback

REQUIREMENTS

• Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field
• Strong background in distributed systems, backend infrastructure, or platform engineering
• Experience building and operating production systems where reliability, latency, and scale are first-class concerns
• Strong sense of developer experience: you think about how systems are used, not just how they work
• Motivated and willing to learn new languages, frameworks, and systems as needed
• Ability to debug complex systems across multiple layers of the stack
• Genuine interest in inference engineering. You don’t need to have hands on experience but are willing to learn
• Excellent communication and collaboration skills

BONUS

• Experience with Kubernetes, including concepts like operators and custom resources
• Prior work on Dynamo, vLLM, SGLang, TensorRT-LLM, or similar inference frameworks
• Experience with distributed scheduling, autoscaling, or service orchestration
• Experience operating GPU workloads in production
• Familiarity with observability tooling, CI/CD systems, or release automation
• Experience contributing to open-source infrastructure or ML systems

BENEFITS

• Competitive compensation, including meaningful equity.
• 100% coverage of medical, dental, and vision insurance for employee and dependents
• Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
• Paid parental leave
• Fertility and family-building stipend through Carrot
• Company-facilitated 401(k)
• Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.

Apply now to embark on a rewarding journey in shaping the future of AI! If you are a motivated individual with a passion for machine learning and a desire to be part of a collaborative and forward-thinking team, we would love to hear from you.

At Baseten, we are committed to fostering a diverse and inclusive workplace. We provide equal employment opportunities to all employees and applicants without regard to race, color, religion, gender, sexual orientation, gender identity or expression, national origin, age, genetic information, disability, or veteran status.

We are an Equal Opportunity Employer and will consider qualified applicants with criminal histories in a manner consistent with applicable law (by example, the requirements of the San Francisco Fair Chance Ordinance, where applicable).