ML Infrastructure Engineer: Scalable Distributed Training

Thinking Machines Lab Inc.; San Francisco (CA); On-site; USD 350,000 - 475,000; Full time

Benefits offered by this job

Health benefits

Dental benefits

Vision benefits

Unlimited PTO

Parental leave

Relocation support

Job summary

Thinking Machines Lab Inc. in San Francisco is seeking an infrastructure research engineer to design and build core systems enabling scalable, efficient training of large models. You will own the training stack end to end to ensure experiments run fast and reliably.

You will work with researchers and engineers to optimize distributed training across thousands of GPUs, develop reusable frameworks, and publish learnings through docs and open-source contributions.

Qualifications

• Bachelor’s degree or equivalent in CS/EE/ML/Physics or similar.
• Strong engineering skills with maintainable code and debugging in complex codebases.
• Understanding of deep learning frameworks (PyTorch, JAX) and their system architectures.
• Thrives in a highly collaborative, cross-functional environment.
• Bias for action to drive across stacks and teams to ship features.

Responsibilities

• Design, implement, and optimize distributed training systems across thousands of GPUs and nodes.
• Develop high-performance optimizations to maximize throughput and efficiency.
• Develop reusable frameworks and libraries to improve training reproducibility, reliability, and scalability.
• Establish standards for reliability, maintainability, and security of systems.
• Collaborate with researchers and engineers to build scalable infrastructure.
• Publish learnings through internal documentation, open-source libraries, or technical reports.

Skills

Distributed systems

Performance optimization

Cross-functional

Code debugging

GPU training

Education

Bachelor's degree

Tools

PyTorch

JAX

Job description

Thinking Machines Lab Inc. in San Francisco is seeking an infrastructure research engineer to design and build core systems enabling scalable, efficient training of large models. You will own the training stack end to end to ensure experiments run fast and reliably.

You will work with researchers and engineers to optimize distributed training across thousands of GPUs, develop reusable frameworks, and publish learnings through docs and open-source contributions.