Senior AI Infra Engineer, Large-Scale ML Systems

Datadog, Inc.

Paris (TX)

Hybrid

USD 180,000 - 240,000

Full time

6 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Stock options
ESPP
Global collaboration across offices
Conference attendance

Job summary

Datadog, Inc. is seeking a Research Engineer for the Frontier Foundations team to build and operate an in-house distributed runtime for large-scale LLM training, inference, and RL workflows.

You will partner with researchers to optimize iteration speed and ensure reliable, scalable runs across GPU clusters. Responsibilities include designing distributed training and inference infrastructure, enabling efficient parallelism, and improving the researcher experience with scheduling and monitoring

Qualifications

  • Experience with distributed systems and ML infrastructure for training and inference at scale.
  • Proficiency in Python and at least one systems language (Rust, C++, or Go).
  • Understanding throughput/latency limits in distributed inference workloads.
  • Strong debugging, profiling, and incident-diagnosis skills.
  • Ability to explain design and performance trade-offs clearly.
  • Prior ML model training experience is valuable but not essential.

Responsibilities

  • Design, build, and operate distributed training and inference infrastructure for large language models.
  • Scale training with parallelism strategies and memory optimization.
  • Build inference and rollout systems for evaluation, experimentation, and RL.
  • Create sandboxed environments for online RL and post-training evaluation.
  • Own performance benchmarks and dashboards to surface regressions.
  • Improve researcher experience: job submission, scheduling, quotas, and GPU utilization.
  • Collaborate with platform teams on cluster capacity and scheduling.
  • Investigate performance and reliability across hardware and software.

Skills

Distributed systems
ML infrastructure
Python
C++
Ray
Kubernetes
Slurm

Tools

Ray
Kubernetes
Slurm

Job description

Datadog, Inc. is seeking a Research Engineer for the Frontier Foundations team to build and operate an in-house distributed runtime for large-scale LLM training, inference, and RL workflows.

You will partner with researchers to optimize iteration speed and ensure reliable, scalable runs across GPU clusters. Responsibilities include designing distributed training and inference infrastructure, enabling efficient parallelism, and improving the researcher experience with scheduling and monitoring

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Infra Engineer: Large-Scale LLM Training
Senior AI Infra Engineer: Large-Scale LLM Training

Datadog, Inc. • Paris (KY)

Hybrid
USD 180,000 - 260,000
RSUs/ESPP
Conferences & meetups
Mentor program
+2
Senior AI GPU Kernel Engineer - High-Performance LLM Training (Hybrid)
Senior AI GPU Kernel Engineer - High-Performance LLM Training (Hybrid)

Datadog, Inc. • Paris (KY)

Hybrid
USD 150,000 - 230,000
Remote ML Infra Engineer for Large-Scale AI Training
Remote ML Infra Engineer for Large-Scale AI Training

true-talent-solutions-partners • San Francisco (CA)

Remote
USD 200,000 - 400,000
Remote-friendly
Competitive salary
Senior AI Research Engineer - GPU Systems & Kernels
Senior AI Research Engineer - GPU Systems & Kernels

Datadog, Inc. • Paris (TX)

On-site
USD 170,000 - 210,000
Senior ML Infra Engineer - Large-Scale Training & Pipelines
Senior ML Infra Engineer - Large-Scale Training & Pipelines

Kindredventures • San Francisco (CA)

On-site
USD 160,000 - 220,000
Senior ML Infra Research Engineer — Distributed Training
Senior ML Infra Research Engineer — Distributed Training

present of AI • San Francisco (CA)

On-site
USD 180,000 - 250,000
Research Engineer, Large-Scale AI Training Infra
Research Engineer, Large-Scale AI Training Infra

ByteDance • San Jose (CA)

On-site
USD 254,000 - 480,000
Senior AI Research Engineer - GPU Systems & Kernels
Senior AI Research Engineer - GPU Systems & Kernels

Datadog, Inc. • Paris (KY)

Hybrid
USD 150,000 - 230,000
Distributed ML Infra Engineer - Large-Scale Pre-Training
Distributed ML Infra Engineer - Large-Scale Pre-Training

Institute of Foundation Models • Sunnyvale (CA)

On-site
USD 150,000 - 450,000
Medical, dental, and vision benefits
Bonus
401K Plan
+4
Senior ML Infra Engineer for Large-Scale Mid-Training & RL
Senior ML Infra Engineer for Large-Scale Mid-Training & RL

Peano AI • Palo Alto (CA)

On-site
USD 200,000 - 260,000