Distributed ML Training Engineer - Scale GPUs, Unlimited PTO

Thinking Machines Lab Inc.; San Francisco, Northern (CA, KY); Hybrid; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. is seeking an infrastructure research engineer to design and build core systems enabling scalable, efficient training of large models for deployment and research. You’ll ensure fast, reliable experimentation and training so research teams can focus on science, not bottlenecks.

This role blends deep systems and performance expertise with curiosity for ML at scale. You’ll own the training stack end to end, driving performance from GPU cycles to scientific progress.

Qualifications

• Bachelor’s degree or equivalent in computer science, electrical engineering, statistics, machine learning, physics, robotics, or similar.
• Strong engineering skills with performant, maintainable code and debugging in complex codebases.
• Understanding of deep learning frameworks (e.g., PyTorch, JAX) and their underlying system architectures.
• Thrive in a highly collaborative environment with cross-functional partners and subject matter experts.
• Bias for action with initiative to work across stacks and teams to ship improvements.

Responsibilities

• Design, implement, and optimize distributed training systems that scale across thousands of GPUs and nodes.
• Develop high-performance optimizations to maximize throughput and efficiency.
• Develop reusable frameworks and libraries to improve training reproducibility, reliability, and scalability for new model architectures.
• Establish standards for reliability, maintainability, and security.
• Collaborate with researchers and engineers to build scalable infrastructure.
• Publish and share learnings through internal docs, open-source libraries, or technical reports.

Skills

Distributed systems

Performance optimization

Team collaboration

Deep learning frameworks (PyTorch/JAX)

Education

Bachelor’s degree or equivalent in CS/EE/ML

Tools

PyTorch

JAX

Megatron-LM

DeepSpeed

XLA

Job description

Thinking Machines Lab Inc. is seeking an infrastructure research engineer to design and build core systems enabling scalable, efficient training of large models for deployment and research. You’ll ensure fast, reliable experimentation and training so research teams can focus on science, not bottlenecks.

This role blends deep systems and performance expertise with curiosity for ML at scale. You’ll own the training stack end to end, driving performance from GPU cycles to scientific progress.