Staff Engineer: Large-Scale ML Training Systems

United States Digital Space LLC; San Francisco (CA); Hybrid; USD 180,000 - 290,000; Full time

Benefits offered by this job

Equity

Travel cost coverage

Hybrid/onsite collaboration

Job summary

Black Forest Labs in San Francisco, with Freiburg ties, is hiring a senior training-systems engineer to optimize production training of large multimodal models. You will work closely with researchers to improve attention performance, kernels, data movement, and training throughput.

You will profile GPU workloads using Nsight tools, validate numerical stability, and help drive low-precision training and quantization paths while ensuring model-quality outcomes.

Qualifications

• Experience working deeply on large-scale training systems, ideally with researchers.
• Strong PyTorch fluency, including reading and modifying low-level training code.
• Experience with distributed training concepts (FSDP, tensor/model parallelism, NCCL).
• Hands-on experience improving throughput, memory footprint, or stability in real training runs.
• Experience profiling GPU workloads with Nsight tools or Torch Profiler.

Responsibilities

• Improve production training performance, reliability, and numerical stability for large multimodal models.
• Profile full training steps across model code, attention, kernels, data loading, and memory pressure.
• Implement GPU-level optimizations: fused kernels, low-precision paths, and quantization kernels.
• Advance FP8/FP4-style training and convergence-safe quantization approaches.
• Collaborate with researchers to translate architecture changes into efficient training implementations.

Skills

PyTorch

Distributed training

Performance profiling

GPU kernels

Low-precision training

Tools

Nsight Systems

Nsight Compute

Torch Profiler

Trace Viewers

Custom Telemetry

Job description

Black Forest Labs in San Francisco, with Freiburg ties, is hiring a senior training-systems engineer to optimize production training of large multimodal models. You will work closely with researchers to improve attention performance, kernels, data movement, and training throughput.

You will profile GPU workloads using Nsight tools, validate numerical stability, and help drive low-precision training and quantization paths while ensuring model-quality outcomes.