Research Engineer: Large-Scale AI Training Systems

BlackForestLabs; San Francisco (CA); Hybrid; USD 180,000 - 290,000; Full time

Benefits offered by this job

Equity

Job summary

Black Forest Labs is recruiting a Member of Technical Staff - Research Engineer to own and optimize large-scale generative-model training systems in a hybrid SF/remote setting. You’ll work closely with researchers to push performance, memory footprint, and stability across GPU clusters.

You’ll implement GPU-level optimizations, profile end-to-end training pipelines, and help translate architectural changes into efficient training code, tooling, and diagnostics for high-quality results.

Qualifications

• Experience working deeply on large-scale training systems, ideally with researchers
• Strong PyTorch fluency, including reading and modifying low-level training code
• Experience with distributed training concepts: FSDP, tensor/model parallelism, NCCL
• Hands-on experience improving throughput, memory footprint, or stability in real runs
• Experience profiling GPU workloads with Nsight tools or torch profiler

Responsibilities

• Improve performance, reliability, and numerical stability of production training runs for large multimodal models
• Profile full training steps across model code, attention, kernels, data loading, encoders, communication, and checkpointing
• Implement and validate GPU-level optimizations: fused kernels, low-precision paths, and quantization kernels
• Debug distributed training failures and build benchmarking/profiling harnesses for trustworthy performance measurements
• Collaborate with researchers to translate architecture changes into efficient training implementations

Skills

PyTorch fluency

Large-scale training

Distributed training

Profiling GPU workloads

Low-precision training

Tools

Nsight Systems

Nsight Compute

Torch profiler

Job description

Black Forest Labs is recruiting a Member of Technical Staff - Research Engineer to own and optimize large-scale generative-model training systems in a hybrid SF/remote setting. You’ll work closely with researchers to push performance, memory footprint, and stability across GPU clusters.

You’ll implement GPU-level optimizations, profile end-to-end training pipelines, and help translate architectural changes into efficient training code, tooling, and diagnostics for high-quality results.