Staff Research Engineer — Large-Scale ML Pre-Training

magic.dev; San Francisco (CA); On-site; USD 275,000 - 550,000; Full time

Benefits offered by this job

Equity

401(k) matching

Health insurance

Unlimited PTO

Visa sponsorship

Relocation stipend

Job summary

Magic is building safe AGI to accelerate humanity's progress. As a Research Engineer on the Pre-training Systems team, you will design and operate the distributed infrastructure that trains Magic’s long-context models at scale.

This role focuses on large-scale model training across GPU clusters, balancing performance, reliability and reproducibility while tackling memory pressure and cross-device communication in production ML systems.

Qualifications

• Strong software engineering and distributed systems fundamentals.
• Experience training large models in multi-node GPU environments.
• Deep understanding of parallelism strategies and performance trade-offs.
• Experience debugging cross-layer issues in production ML systems.
• Strong ownership mindset and ability to operate critical infrastructure.
• Track record of improving performance or reliability of large-scale systems.

Responsibilities

• Scale distributed training across large GPU clusters.
• Optimize communication patterns and gradient synchronization.
• Improve checkpointing, fault tolerance, and job recovery systems.
• Profile and eliminate performance bottlenecks across compute, networking, and storage.
• Improve experiment reproducibility and orchestration workflows.
• Increase hardware utilization and training throughput.
• Collaborate with Kernels and Research to align model architecture with systems realities.

Skills

Software engineering

Distributed systems

Large-model training

Performance debugging

Job description

Magic is building safe AGI to accelerate humanity's progress. As a Research Engineer on the Pre-training Systems team, you will design and operate the distributed infrastructure that trains Magic’s long-context models at scale.

This role focuses on large-scale model training across GPU clusters, balancing performance, reliability and reproducibility while tackling memory pressure and cross-device communication in production ML systems.