ML Training Infra Engineer for Autonomous AI

Lightyearz; Palo Alto (CA); Hybrid; USD 104,000 - 240,000; Full time

Benefits offered by this job

Medical plan with $0 payroll deduction

Pension plans

Sick and Vacation time

Employee discounts

Paid Holidays

Job summary

Lightyearz is building autonomous AI capabilities and scalable neural network training infrastructure. As a Software Engineer, you will reinforce and optimize our training and auto-labeling systems across large GPU clusters and cloud infrastructure.

You will implement robust Python/C++ code, integrate with CI, profile performance, and collaborate with the hardware team to keep jobs running smoothly at scale.

Qualifications

• Practical experience programming in Python and/or C/C++.
• Experience with training frameworks, ideally PyTorch.
• Demonstrated experience scaling neural network training across clusters of GPUs.
• Understanding of modern machine learning concepts and deep learning.
• Optional: Devops experience with training clusters and large data filesystems.

Responsibilities

• Write robust Python software in the ML training repository with best practices to support scientists.
• Integrate training software into CI to persist metrics and manage neural network builds.
• Profile and optimize performance to reduce convergence time across CPU/GPU nodes.
• Coordinate with hardware cluster team to maintain high availability and throughput.

Skills

Python

C/C++

PyTorch

GPU clusters

System-level software

CUDA

Job description

Lightyearz is building autonomous AI capabilities and scalable neural network training infrastructure. As a Software Engineer, you will reinforce and optimize our training and auto-labeling systems across large GPU clusters and cloud infrastructure.

You will implement robust Python/C++ code, integrate with CI, profile performance, and collaborate with the hardware team to keep jobs running smoothly at scale.