ML Performance Engineer — Distributed Training & Kernels

Veeda

California (MO)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Veeda AI is seeking a Member of Technical Staff - ML Performance to optimize distributed training for multi-node video world models. You will own step time, FLOPs utilization, memory patterns, and parallelism strategies across NVLink, NCCL, and PyTorch FSDP2 in real workloads.

You’ll implement kernels, tune CUDA/Triton components, improve precision stability, and contribute to fault diagnostics and elastic checkpointing.

Qualifications

Bachelor's degree or equivalent hands-on experience in Computer Science, Computer Engineering, or a related technical field.
Deep hands-on experience with PyTorch and at least one large-scale parallelism stack on real multi-node jobs.
Fluency in Python and C++/CUDA with the ability to predict where a kernel will stall from its memory access pattern before profiling.
Experience profiling live training runs with Nsight Systems or the PyTorch profiler and translating traces into quantifiable step-time or MFU improvements.
Expertise in at least one of low-precision numerics, kernel authoring, or large-run fault diagnosis, and credibility in the others.

Responsibilities

Own step time and model FLOPs utilization for multi-node video world model training, choosing the tensor, context, and expert parallelism mix in PyTorch FSDP2 and Megatron-Core rather than inheriting a default.
Take BF16, FP8, and NVFP4 recipes from running to converging on Blackwell, chasing scaling-factor and accumulation bugs into the video tokenizer and VAE layers where the activation outliers actually live.
Write and tune the CUDA and Triton kernels PyTorch does not give us, driving FlashAttention-4, FlexAttention, and torch.compile integration so quadratic attention over long video sequences stops setting step time.
Tune NCCL collectives and compute/communication overlap across NVLink domains and the fabric, using the NCCL flight recorder to turn a watchdog timeout into a named rank and collective, not a restart.
Build the detection layer for silent data corruption (SDC), stuck CUDA kernels, and "card-freeze" hangs, plus asynchronous and tiered checkpointing that makes an interruption cost minutes rather than a day.