Senior AI Infra Engineer: Large-Scale LLM Training

Datadog, Inc.

Paris (KY)

Hybrid

USD 180,000 - 260,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

RSUs/ESPP
Conferences & meetups
Mentor program
Employee resource groups
Global benefits

Job summary

Datadog is seeking a Research Engineer on the Frontier Foundations team to partner with researchers and build in-house distributed runtime for large-scale LLM training, RL, and inference. You will own how researchers iterate and how long training runs complete, across GPU clusters.

The role sits within the Datadog AI Research initiative, focused on scalable infrastructure and high-risk, high-reward problems in cloud observability and security.

Qualifications

  • Depth in distributed systems and ML infrastructure for training and inference at scale.
  • Proficient in Python and at least one systems language (Rust, C++, or Go).
  • Experience operating large-scale training or serving infrastructures, schedulers, or orchestration systems for GPU workloads.
  • Understanding of throughput/latency limits in distributed inference: communication, memory, scheduling, stragglers.
  • Strong debugging, profiling, and incident-diagnosis skills; on-call experience preferred.
  • Experience training machine learning models is valuable but not essential for engineers with distributed systems/HPC backgrounds.

Responsibilities

  • Design, build, and operate distributed training and inference infrastructure for large language models, including Mixture-of-Experts models.
  • Scale training with parallelism strategies and memory optimization.
  • Build inference and rollout systems that support evaluation, experimentation, and RL without slowing down training.
  • Create execution environments and sandboxes for online RL and post-training evaluation.
  • Own performance validation with dashboards to surface regressions in rollout speed and training throughput.
  • Improve researcher experience: job submission, scheduling, quotas, GPU utilization, and observability.
  • Collaborate with platform teams on cluster capacity, networking, and GPU provider scheduling.
  • Investigate performance and reliability issues across hardware and software.

Skills

Distributed systems
ML infrastructure
Python
Rust
C++
Ray
Kubernetes
Schedulers
GPU clusters
LLM training

Tools

Megatron-LM
TorchTitan
DeepSpeed
vLLM
SGLang
NCCL
RDMA

Job description

Datadog is seeking a Research Engineer on the Frontier Foundations team to partner with researchers and build in-house distributed runtime for large-scale LLM training, RL, and inference. You will own how researchers iterate and how long training runs complete, across GPU clusters.

The role sits within the Datadog AI Research initiative, focused on scalable infrastructure and high-risk, high-reward problems in cloud observability and security.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Infra Engineer, Large-Scale ML Systems
Senior AI Infra Engineer, Large-Scale ML Systems

Datadog, Inc. • Paris (TX)

Hybrid
USD 180,000 - 240,000
Stock options
ESPP
Global collaboration across offices
+1
Senior AI GPU Kernel Engineer - High-Performance LLM Training (Hybrid)
Senior AI GPU Kernel Engineer - High-Performance LLM Training (Hybrid)

Datadog, Inc. • Paris (KY)

Hybrid
USD 150,000 - 230,000
Senior AI Research Engineer - GPU Systems & Kernels
Senior AI Research Engineer - GPU Systems & Kernels

Datadog, Inc. • Paris (TX)

On-site
USD 170,000 - 210,000
Senior AI Research Engineer - LLM Training & Inference Infrastructure
Senior AI Research Engineer - LLM Training & Inference Infrastructure

Datadog, Inc. • Paris (KY)

Hybrid
USD 180,000 - 260,000
RSUs/ESPP
Conferences & meetups
Mentor program
+2
Senior AI Research Engineer - GPU Systems & Kernels
Senior AI Research Engineer - GPU Systems & Kernels

Datadog, Inc. • Paris (KY)

Hybrid
USD 150,000 - 230,000
Senior ML Infra Engineer for Large-Scale Mid-Training & RL
Senior ML Infra Engineer for Large-Scale Mid-Training & RL

Peano AI • Palo Alto (CA)

On-site
USD 200,000 - 260,000
Remote ML Infra Engineer for Large-Scale AI Training
Remote ML Infra Engineer for Large-Scale AI Training

true-talent-solutions-partners • San Francisco (CA)

Remote
USD 200,000 - 400,000
Remote-friendly
Competitive salary
Senior ML Infra Engineer - Large-Scale Training & Pipelines
Senior ML Infra Engineer - Large-Scale Training & Pipelines

Kindredventures • San Francisco (CA)

On-site
USD 160,000 - 220,000
Research Engineer, Large-Scale AI Training Infra
Research Engineer, Large-Scale AI Training Infra

ByteDance • San Jose (CA)

On-site
USD 254,000 - 480,000
AI Research Engineer, LLMs and Large Scale Training
AI Research Engineer, LLMs and Large Scale Training

Harnham • United States

Hybrid
USD 180,000 - 240,000