AI Training Systems Architect (Distributed)

Unconventional AI; Palo Alto (CA); On-site; USD 210,000 - 320,000; Full time

Benefits offered by this job

Health benefits

401k matching

Unlimited PTO

Complimentary meals in Palo Alto

Job summary

Unconventional AI in Palo Alto seeks a senior ML systems engineer to design and build the next-generation training platform. You will co-design and implement training ecosystems alongside novel AI models and hardware platforms that push the boundaries of physics-based compute.

You will optimize multi-node distributed training, implement elastic sharding, and develop robust data streaming and checkpointing, with a focus on CUDA/Triton kernels and MFU benchmarking.

Qualifications

• MS/PhD or equivalent in AI/ML, CS, Physics, Electrical Engineering, or Applied Math.
• Experience mapping state-of-the-art AI model architectures to system performance with distributed training and parallelism.
• Proven track record implementing production-grade training frameworks (Megatron-LM, DeepSpeed, Ray, PyTorch Lightning).

Responsibilities

• Build and maintain highly optimized, model-specific training stacks for Gen Vision, language, and world models.
• Design and scale multi-node distributed training with elastic sharding and robust data streaming pipelines.
• Develop and optimize kernels using CUDA and Triton; design benchmarking for MFU, memory bandwidth, and convergence.
• Translate model requirements into concrete infrastructure and hardware specs for cross-functional teams.

Job description

Unconventional AI in Palo Alto seeks a senior ML systems engineer to design and build the next-generation training platform. You will co-design and implement training ecosystems alongside novel AI models and hardware platforms that push the boundaries of physics-based compute.

You will optimize multi-node distributed training, implement elastic sharding, and develop robust data streaming and checkpointing, with a focus on CUDA/Triton kernels and MFU benchmarking.