Distributed Training Infra Engineer for Large Models

Kindredventures; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Kindredventures in San Francisco is seeking an infrastructure engineer to scale distributed training for Large Physics models. You will design, implement, and optimize systems that run thousands of GPUs and accelerate research progress.

Collaborate with researchers to bring prototype models to full scale, optimize memory and throughput, and contribute to open-source ML infrastructure. You should have strong expertise in PyTorch and JAX and a track record of performance profiling.

Qualifications

• Experience with distributed training frameworks and techniques to train large foundation models.
• Strong grasp of parallelism, memory optimization, mixed precision, and communication overlap.
• Ability to profile and debug performance in complex codebases from framework internals down to kernels and collectives.
• Deep understanding of PyTorch and JAX and their underlying system architectures.
• Bonus: contributions to open-source ML infrastructure (e.g. PyTorch, Megatron-LM, DeepSpeed, XLA).

Responsibilities

• Design, implement, and optimize distributed training systems that scale across thousands of GPUs
• Research and test parallelization strategies and numerical precision trade-offs across model scales
• Analyze, profile, and debug low-level GPU operations to maximize throughput and hardware utilization
• Build reusable frameworks for checkpointing, fault tolerance, and reproducibility that stay robust under rapid research iteration
• Collaborate with researchers to bring novel model architectures from prototype to full scale
• Stay up-to-date on research to bring new ideas to work

Skills

Distributed training

FSDP

DeepSpeed

Megatron

PyTorch

JAX/XLA

Performance profiling

Memory optimization

Mixed precision

Kernels/Collectives

Open-source contributions

Tools

N/A

Job description

Kindredventures in San Francisco is seeking an infrastructure engineer to scale distributed training for Large Physics models. You will design, implement, and optimize systems that run thousands of GPUs and accelerate research progress.

Collaborate with researchers to bring prototype models to full scale, optimize memory and throughput, and contribute to open-source ML infrastructure. You should have strong expertise in PyTorch and JAX and a track record of performance profiling.