Staff Research Engineer, Large-Scale Pretraining Systems

Magic; San Francisco (CA); On-site; USD 275,000 - 550,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation stipend

Equity

Job summary

Magic in San Francisco is seeking a Member of Technical Staff for the Pretraining Systems team to design and operate the distributed infrastructure that trains its long-context models at scale.

You will work at the boundary of deep learning and distributed systems, tackling memory pressure, cross-device communication, fault-tolerant long-running jobs, and efficient sequence packing to keep pretraining fast and reliable.

Qualifications

• Strong software engineering and distributed systems fundamentals.
• Experience training large models in multi-node GPU environments.
• Deep understanding of parallelism strategies and performance trade-offs.
• Experience debugging cross-layer issues in production ML systems.
• Strong ownership mindset and ability to operate critical infrastructure.
• Track record of improving performance or reliability of large-scale systems.

Responsibilities

• Scale distributed training across large GPU clusters (data, tensor, pipeline parallelism)
• Optimize communication patterns and gradient synchronization
• Improve checkpointing, fault tolerance, and job recovery systems
• Profile and eliminate performance bottlenecks across compute, networking, and storage
• Improve experiment reproducibility and orchestration workflows
• Increase hardware utilization and training throughput
• Collaborate with Kernels and Research to align model architecture with systems realities

Skills

Distributed systems

Large-model training

Parallelism strategies

Performance optimization

Production ML systems

Tools

CUDA

PyTorch

TensorFlow

Job description

Magic in San Francisco is seeking a Member of Technical Staff for the Pretraining Systems team to design and operate the distributed infrastructure that trains its long-context models at scale.

You will work at the boundary of deep learning and distributed systems, tackling memory pressure, cross-device communication, fault-tolerant long-running jobs, and efficient sequence packing to keep pretraining fast and reliable.