Software Engineer - Training Infrastructure

present of AI; San Francisco (CA); On-site; USD 190,000 - 260,000; Full time

Job summary

Baseten is seeking a Software Engineer for Training Infrastructure in San Francisco to own core ML training platform systems, including scheduling, storage, and observability. A strong candidate has 5+ years of experience, solid Go skills, and hands-on Kubernetes in production.

The role emphasizes distributed systems design, tuning for performance, and mentoring junior engineers as the team scales with reinforcement learning workloads.

Qualifications

• 5+ years of experience in software engineering or a similar role.
• Strong Go programming skills with production experience.
• Hands-on experience running Kubernetes at scale in production.

Responsibilities

• Design, own, and operate the core systems behind a machine learning training platform (scheduling, storage, networking, reliability, observability).
• Mentor and guide other engineers to balance speed and stability while improving the platform.

Skills

Go

Distributed systems

Mentoring

Observability

Tools

Kubernetes

Job description

Software Engineer - Training Infrastructure

This role centers on designing and owning the core systems behind a machine learning training platform, including scheduling, storage, networking, reliability, and observability. A strong candidate has at least five years of experience, solid Go skills, and deep hands-on Kubernetes work in production. They also bring a good grasp of distributed systems and tuning, and the judgment to weigh speed against stability while mentoring less experienced engineers.

How it fits the AI era

Training and reinforcement learning workloads are growing fast, and the platform that runs them has to keep up with large, uneven demand for compute. AI handles none of the hard tradeoffs here, so the human work is deciding how jobs get scheduled across regions, how failures are caught, and how researchers can trust the system. Engineers who understand what model developers actually need will turn that into dependable infrastructure.

How it builds AI experience

The work sits close to real model training, including reinforcement learning systems and continuous learning pipelines, built for customers who deploy models on an inference platform. Someone in the role would learn how open source training tools behave at scale and how research engineers use them. To prepare, try running distributed training jobs with PyTorch and a tool such as DeepSpeed or FSDP, and notice where they fail or slow down.

Worth showing

• Go services in production
• Kubernetes at scale
• Observability system design
• Distributed training experience

This is our own read, written from the employer's posting and the company's Spotlight. The employer's posting is the source of truth for the role.

About Baseten

High-performance cloud inference platform for deploying, serving, and scaling open-source and custom artificial intelligence models.