Training Infrastructure Engineer - Scale Large-Scale AI

Fireworks AI; New York (NY); On-site; USD 170,000 - 230,000; Full time

Job summary

Fireworks AI is seeking a Training Infrastructure Engineer to design, build, and optimize the infrastructure that powers our large-scale model training operations. You will collaborate with AI researchers to create robust training pipelines and ensure efficient, scalable training across GPUs and data centers.

You will implement monitoring, logging, and automation to streamline provisioning and orchestration, while advancing training methodologies and reducing costs.

Qualifications

• Experience building scalable ML training infrastructure.
• Knowledge of distributed training techniques (data parallelism, model parallelism, FSDP).
• Experience with cloud platforms and container orchestration tools.

Responsibilities

• Design and implement scalable infrastructure for large-scale model training workloads
• Develop and maintain distributed training pipelines for LLMs and multimodal models
• Optimize training performance across multiple GPUs, nodes, and data centers
• Implement monitoring, logging, and debugging tools for training operations
• Architect and maintain data storage solutions for large-scale training datasets
• Automate infrastructure provisioning, scaling, and orchestration for model training
• Collaborate with researchers to implement and optimize training methodologies
• Analyze and improve efficiency, scalability, and cost-effectiveness of training systems
• Troubleshoot complex performance issues in distributed training environments

Skills

Distributed systems

ML infrastructure

PyTorch

Cloud platforms

Kubernetes

Docker

Data parallelism

Model parallelism

FSDP

Performance optimization

Education

Bachelor's degree in CS/Engineering or equivalent

Master's degree preferred

Tools

PyTorch

AWS

GCP

Azure

Kubernetes

Docker

MLflow

Job description

Fireworks AI is seeking a Training Infrastructure Engineer to design, build, and optimize the infrastructure that powers our large-scale model training operations. You will collaborate with AI researchers to create robust training pipelines and ensure efficient, scalable training across GPUs and data centers.

You will implement monitoring, logging, and automation to streamline provisioning and orchestration, while advancing training methodologies and reducing costs.