Staff ML Infra Engineer: Distributed Training & Inference

Jobtailor; Boston (MA); On-site; USD 120,000 - 160,000; Full time

Job summary

Core AI is seeking an experienced infrastructure engineer to own the training and inference platform. You will run distributed multi-GPU training jobs, GPU scheduling, and model-serving systems (vLLM, SGLang, Triton) across proprietary models and self-hosted inference.

You will build tooling researchers rely on to launch training runs, route workloads, and ensure capacity, observability, and reliability. The role involves debugging under real load and working hands-on, turning research code into

Qualifications

• Three or more years building and operating ML training or inference infrastructure in production.
• Hands-on experience with distributed training (multi-GPU or multi-node) and model-serving systems.
• Strong software engineering fundamentals for reliable, scalable services.
• ML fluency to debug training loops and inference-time behavior with researchers.

Responsibilities

• Own the training and inference infrastructure for Core AI, including distributed training jobs, GPU scheduling, and model-serving systems.
• Develop tools and abstractions for researchers to launch training runs and route workloads across models.
• Partner with Engineering on capacity planning, observability, and reliability for GPU and inference infrastructure.
• Debug and harden the training and inference stack under real load.
• Stay hands-on by writing code and leading when a training job stalls or an inference path breaks.

Tools

PyTorch

Ray

vLLM

SGLang

Triton

Terraform

CUDA

GCP

AWS

Job description

Core AI is seeking an experienced infrastructure engineer to own the training and inference platform. You will run distributed multi-GPU training jobs, GPU scheduling, and model-serving systems (vLLM, SGLang, Triton) across proprietary models and self-hosted inference.

You will build tooling researchers rely on to launch training runs, route workloads, and ensure capacity, observability, and reliability. The role involves debugging under real load and working hands-on, turning research code into