Staff Research Engineer: Large-Scale AI Training

Black Forest Labs Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 180,000 - 290,000; Full time

Job summary

Black Forest Labs Inc. is seeking an experienced engineer to join our production training effort for large multimodal models. You will work closely with researchers to optimize training throughput, memory usage, and numerical stability across GPUs and large GPU fleets.

The role emphasizes low-level training code, profiling, and kernel development, with opportunities to influence architecture-level training choices. Base salary ranges for US candidates are highlighted in the ad.

Qualifications

• Experience with large-scale training systems, preferably with researchers
• Strong PyTorch fluency and low-level training code familiarity
• Understanding distributed training concepts (FSDP, tensor/model parallelism, NCCL)
• Hands-on improvement of training throughput, memory footprint or stability
• Experience profiling GPU workloads with Nsight or similar tools
• Ability to verify numerical behavior and ownership of results
• Knowledge of FP8/FP4-precision training and quantization tradeoffs
• Comfort operating in ambiguous, fast-moving research-to-production environments

Responsibilities

• Improve performance, reliability, and stability of production training runs for large multimodal models
• Profile training steps across model code, attention, kernels, data loading, and memory
• Implement and validate GPU optimizations: fused kernels, low-precision paths, quantization kernels
• Advance FP8/FP4-style training and quantization strategies
• Translate architecture changes into efficient training implementations with researchers
• Debug distributed training issues (NaNs, memory leaks, NCCL, throughput)
• Build benchmarking and profiling tools for cross-hardware validation
• Collaborate with researchers and training team to remove bottlenecks

Skills

PyTorch

Distributed training

Profiling tools

Low-precision training

GPU kernels

NCCL

Tools

Nsight Systems

Nsight Compute

CUDA

Triton

CUTLASS

MXFP8/FP8

Job description

Black Forest Labs Inc. is seeking an experienced engineer to join our production training effort for large multimodal models. You will work closely with researchers to optimize training throughput, memory usage, and numerical stability across GPUs and large GPU fleets.

The role emphasizes low-level training code, profiling, and kernel development, with opportunities to influence architecture-level training choices. Base salary ranges for US candidates are highlighted in the ad.