AI Training Infrastructure Engineer – Distributed GPU

Fdetimes; San Mateo (CA); On-site; USD 140,000 - 230,000; Full time

Job summary

Fireworks AI seeks a Training Infrastructure Engineer to design, build, and optimize the infrastructure powering our large-scale model training operations. Your work will be essential to developing high-performance AI training pipelines in collaboration with researchers and engineers.

Design scalable infrastructure for LLM and multimodal training, optimize performance across GPUs, nodes, and data centers, and implement monitoring, logging, and automation.

Qualifications

• Bachelor's degree or equivalent practical experience.
• 3+ years in distributed systems and ML infrastructure.
• Experience with PyTorch.
• Experience with cloud platforms (AWS, GCP, Azure).
• Experience with containers and orchestration (Kubernetes, Docker).
• Knowledge of distributed training techniques.

Responsibilities

• Design scalable infrastructure for large-scale model training workloads.
• Develop and maintain distributed training pipelines for LLMs and multimodal models.
• Optimize training performance across GPUs, nodes, and data centers.
• Implement monitoring, logging, and debugging tools for training operations.
• Architect and maintain data storage solutions for large-scale training datasets.
• Automate infrastructure provisioning, scaling, and orchestration.
• Collaborate with researchers to implement and optimize training methodologies.
• Analyze and improve efficiency, scalability, and cost-effectiveness of training systems.
• Troubleshoot complex performance issues in distributed training environments.

Skills

Distributed systems

ML infrastructure

Cloud platforms

Education

Bachelor's degree in CS/related field

Tools

PyTorch

Kubernetes

Docker

AWS

GCP

Azure

FSDP

Job description

Fireworks AI seeks a Training Infrastructure Engineer to design, build, and optimize the infrastructure powering our large-scale model training operations. Your work will be essential to developing high-performance AI training pipelines in collaboration with researchers and engineers.

Design scalable infrastructure for LLM and multimodal training, optimize performance across GPUs, nodes, and data centers, and implement monitoring, logging, and automation.