Senior Research Engineer, Large-Scale ML Pre-training

Magic; San Francisco (CA); On-site; USD 275,000 - 474,000; Full time

Benefits offered by this job

Equity in total compensation

401(k) with matching

Health, dental and vision insurance

Unlimited PTO

Job summary

Magic is seeking a Research Engineer for the Pre-training Systems team to design and operate the distributed infrastructure that trains its long-context models at scale.

You will own systems ensuring training runs are performant, reliable, and reproducible across massive GPU clusters, addressing memory pressure, communication overhead, and fault tolerance while pushing training throughput.

Qualifications

• Strong software engineering fundamentals and distributed systems knowledge.
• Experience training large models on multi-node GPU environments.
• Proficiency in debugging cross-layer issues in production ML systems.

Responsibilities

• Scale distributed training across large GPU clusters.
• Optimize communication patterns and gradient synchronization.
• Improve checkpointing, fault tolerance, and job recovery systems.
• Profile and remove performance bottlenecks across compute, networking, storage.

Skills

Software engineering

Distributed systems

Multi-node training

Performance debugging

Ownership mindset

Job description

Magic is seeking a Research Engineer for the Pre-training Systems team to design and operate the distributed infrastructure that trains its long-context models at scale.

You will own systems ensuring training runs are performant, reliable, and reproducible across massive GPU clusters, addressing memory pressure, communication overhead, and fault tolerance while pushing training throughput.