Member of Technical Staff, Pretraining Systems

Magic; San Francisco (CA); On-site; USD 275,000 - 550,000; Full time

Benefits offered by this job

Visa sponsorship

Relocation stipend

Equity

Job summary

Magic in San Francisco is seeking a Member of Technical Staff for the Pretraining Systems team to design and operate the distributed infrastructure that trains its long-context models at scale.

You will work at the boundary of deep learning and distributed systems, tackling memory pressure, cross-device communication, fault-tolerant long-running jobs, and efficient sequence packing to keep pretraining fast and reliable.

Qualifications

• Strong software engineering and distributed systems fundamentals.
• Experience training large models in multi-node GPU environments.
• Deep understanding of parallelism strategies and performance trade-offs.
• Experience debugging cross-layer issues in production ML systems.
• Strong ownership mindset and ability to operate critical infrastructure.
• Track record of improving performance or reliability of large-scale systems.

Responsibilities

• Scale distributed training across large GPU clusters (data, tensor, pipeline parallelism)
• Optimize communication patterns and gradient synchronization
• Improve checkpointing, fault tolerance, and job recovery systems
• Profile and eliminate performance bottlenecks across compute, networking, and storage
• Improve experiment reproducibility and orchestration workflows
• Increase hardware utilization and training throughput
• Collaborate with Kernels and Research to align model architecture with systems realities

Skills

Distributed systems

Large-model training

Parallelism strategies

Performance optimization

Production ML systems

Tools

CUDA

PyTorch

TensorFlow

Job description

Member of Technical Staff, Pretraining Systems

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pretraining, domain-specific RL, ultra-long context, and inference-time compute to achieve this goal.

About the role

As a Research Engineer on the Pretraining Systems team, you will design and operate the distributed infrastructure that trains Magic’s long-context models at scale.

This role focuses on large-scale model training across massive GPU clusters. You will work at the boundary between deep learning and distributed systems, ensuring that training runs are performant, reliable, and reproducible under extreme scale.

Magic’s long-context models create non-trivial systems challenges: sustained memory pressure, communication overhead across thousands of devices, long-running jobs that must survive failures, and efficient sequence packing under hardware constraints. You will own the systems that make large-scale pretraining stable and fast.

What you’ll work on

Scale distributed training across large GPU clusters (data, tensor, pipeline parallelism)

Optimize communication patterns and gradient synchronization

Improve checkpointing, fault tolerance, and job recovery systems

Profile and eliminate performance bottlenecks across compute, networking, and storage

Improve experiment reproducibility and orchestration workflows

Increase hardware utilization and training throughput

Collaborate with Kernels and Research to align model architecture with systems realities

What we’re looking for

Strong software engineering and distributed systems fundamentals

Experience training large models in multi-node GPU environments

Deep understanding of parallelism strategies and performance trade-offs

Experience debugging cross-layer issues in production ML systems

Strong ownership mindset and ability to operate critical infrastructure

Track record of improving performance or reliability of large-scale systems

Our culture

Integrity. Words and actions should be aligned

Hands-on. At Magic, everyone is building

Teamwork. We move as one team, not N individuals

Focus. Safely deploy AGI. Everything else is noise

Quality. Magic should feel like magic

Magic strives to be the place where high-potential individuals can do their best work. We value quick learning and grit just as much as skill and experience.

Compensation, benefits, and perks (US):

Annual salary range: $275K - $550K

Equity is a significant part of total compensation, in addition to salary

401(k) plan with 6% salary matching

Generous health, dental and vision insurance for you and your dependents

Unlimited paid time off

Visa sponsorship and relocation stipend to bring you to SF, if possible

A small, fast-paced, highly focused team