Scale Low-Latency ML Inference Engineer (GPU/CUDA)

AI Breaking Wire; San Francisco (CA); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Equity

Medical benefits (full coverage)

PTO & Hybrid work policy

Lunch & wellness stipends

Job summary

AI Breaking Wire seeks a Machine Learning Engineer to join our Inference Infrastructure team. You will build and optimize the high-throughput, low-latency distributed systems that power models like GPT-4 and Sora for millions of users worldwide.

The role emphasizes C++ and Python proficiency, GPU optimization with CUDA and Triton, and collaboration with research teams to productionize new architectures. This hybrid position is based in San Francisco.

Qualifications

• BS/MS/PhD in Computer Science or related technical field.
• 3+ years of industry experience building large-scale distributed systems or ML infrastructure.
• Deep proficiency in C++ and Python.
• Extensive experience with CUDA, Triton, or deep learning hardware accelerators.
• Familiarity with distributed training and inference frameworks (vLLM, TensorRT-LLM, Megatron).

Responsibilities

• Architect, build, and scale low-latency distributed inference serving systems for massive generative models.
• Optimize GPU utilization, memory management, and kernel execution for state-of-the-art transformer architectures.
• Collaborate with research teams to ensure smooth transition of new model architectures into production.
• Monitor system performance, troubleshoot bottlenecks, and implement robust reliability measures.

Skills

C++

Python

CUDA

Triton

Megatron

Education

BS/MS/PhD in Computer Science or related field

Tools

vLLM

TensorRT-LLM

Megatron

Job description

AI Breaking Wire seeks a Machine Learning Engineer to join our Inference Infrastructure team. You will build and optimize the high-throughput, low-latency distributed systems that power models like GPT-4 and Sora for millions of users worldwide.

The role emphasizes C++ and Python proficiency, GPU optimization with CUDA and Triton, and collaboration with research teams to productionize new architectures. This hybrid position is based in San Francisco.