RL Systems Engineer: Scalable GPU Training & Inference

Collision AI; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Collision AI is building models and agents for cyber defense. You’ll develop infrastructure that trains them through reinforcement learning on multi-GPU clusters.

The role blends systems engineering with practical RL, ensuring training correctness while improving performance at scale. You will collaborate with researchers to diagnose issues across trainer and inference workloads, implement robust debugging workflows, and contribute to reproducible experimentation during long-running training

Qualifications

• Strong Python and PyTorch skills.
• Experience building distributed training or inference infrastructure.
• Practical understanding of RL in LLM contexts.
• Experience profiling GPU workloads and resolving ML system issues.
• Ability to investigate problems across research and infrastructure code.

Responsibilities

• Coordinate trainers, rollout workers, environment execution, and reward computation across GPU clusters.
• Improve distributed training with FSDP, tensor/pipeline parallelism, and NCCL.
• Develop monitoring and diagnostics for stale weights and drift during long runs.

Skills

Python

PyTorch

Distributed training

Reinforcement learning infrastructure

GPU profiling

Asynchronous/off-policy training

SGLang

vLLM

NCCL

CUDA/Triton

Tools

SGLang

vLLM

NCCL

CUDA

Triton

Job description

Collision AI is building models and agents for cyber defense. You’ll develop infrastructure that trains them through reinforcement learning on multi-GPU clusters.

The role blends systems engineering with practical RL, ensuring training correctness while improving performance at scale. You will collaborate with researchers to diagnose issues across trainer and inference workloads, implement robust debugging workflows, and contribute to reproducible experimentation during long-running training