Distributed Systems Engineer - LLM Inference Platform

US Health Partners, LLC; New York (NY); Hybrid; USD 150,000 - 210,000; Full time

Benefits offered by this job

Equity and competitiveness

US benefits: medical/dental/vision for

Flexible PTO

Parental leave

Family-building stipend

401(k)

Job summary

Baseten is hiring distributed systems engineers to build the runtime for large-scale LLM inference, focusing on speed, reliability, and cost efficiency. You will work on Model APIs and the orchestration layers that deploy ML models at scale, ranging from developer UX to the execution engines behind tool calls and multimodal serving.

The role emphasizes ownership of production systems, cross-functional collaboration, and a bias toward practical, observable performance improvements in a

Qualifications

• 3+ years building and operating distributed systems or large-scale APIs.
• Proven track record of low-latency, reliable backend services.
• Profiling, tracing, capacity planning, and SLO management experience.
• Ability to debug performance across stack layers and runtime internals.
• Strong focus on developer experience and usability.
• Willingness to learn new languages, frameworks, and systems.
• Experience with ML or model serving is a plus.
• Excellent written communication and collaboration skills.

Responsibilities

• Build the infrastructure and orchestration systems for large-scale LLM inference.
• Design, build, and operate Model APIs with JSON mode and tool calling.
• Implement API versioning, validation, usage metering, quotas, and auth.
• Instrument observability and run benchmarks for speed and reliability.
• Debug production systems spanning Kubernetes, runtimes, and GPU workloads.
• Partner with Inference Performance engineers to optimize features.
• Own projects end-to-end from architecture to deployment and iteration.

Skills

Distributed systems

Backend infrastructure

Low latency

Developer experience

Performance debugging

Cross-functional collaboration

ML/LLM awareness

Education

Bachelor's/Master's/PhD in CS or Engineering

Tools

Kubernetes

Observability tooling

CI/CD

API gateways

Job description

Baseten is hiring distributed systems engineers to build the runtime for large-scale LLM inference, focusing on speed, reliability, and cost efficiency. You will work on Model APIs and the orchestration layers that deploy ML models at scale, ranging from developer UX to the execution engines behind tool calls and multimodal serving.

The role emphasizes ownership of production systems, cross-functional collaboration, and a bias toward practical, observable performance improvements in a