Distributed ML Training Platform Engineer

Remote; San Francisco (CA); Presencial; USD 180.000 - 260.000; Jornada completa

Descripción de la vacante

Remote in San Francisco seeks an engineer to own and optimize a GPU training platform and large-scale model training workloads on a GKE-based cluster. You will collaborate with a healthcare and life-sciences AI client and researchers to tune, benchmark, and debug training jobs end-to-end.

The role focuses on training workloads, profiling, and ensuring performance and reliability, with close partnership with the client’s scientists and engineers. You will own your area and drive design decisions.

Formación

• 5+ years of engineering experience, with recent hands-on work on GPU training infrastructure and large-scale training.
• Strong Python software engineering: production-quality code, tests and code review.
• GKE and GPU scheduling on Google Cloud with GPU node pools, job queueing and capacity management.
• Distributed PyTorch and Ray Train: hands-on multi-node training experience.

Responsabilidades

• Own and optimize the training platform, diagnose and fix root causes, and work with infrastructure and scientists.
• Run, scale and debug multi-node distributed training with PyTorch and Ray Train.
• Profile jobs to identify bottlenecks across compute, memory, and interconnect, and tune RDMA/GPU networking.
• Instrument training pipelines with telemetry, maintain end-to-end tests, and ensure reliability.

Conocimientos

Python

Shell scripting

Software engineering

Herramientas

Gemini

GPT-4

LLaMA

Vertex AI Vector Search

pgvector

NCCL

RDMA

Descripción del empleo

Remote in San Francisco seeks an engineer to own and optimize a GPU training platform and large-scale model training workloads on a GKE-based cluster. You will collaborate with a healthcare and life-sciences AI client and researchers to tune, benchmark, and debug training jobs end-to-end.

The role focuses on training workloads, profiling, and ensuring performance and reliability, with close partnership with the client’s scientists and engineers. You will own your area and drive design decisions.