Senior ML Performance Engineer - Distributed Training

Veeda AI; Toronto; On-site; CAD 120,000 - 180,000; Full time

Job summary

Veeda AI in Toronto is seeking a Member of Technical Staff - ML Performance to own step time and model throughput for multi-node video world model training, optimize tensor usage, and implement advanced parallelism.

You will write and tune CUDA and Triton kernels, improve precision and numerical stability, and build fault diagnostics and elastic checkpointing to minimize downtime during interruptions.

Qualifications

• Bachelor's degree or equivalent hands-on experience in Computer Science, Computer Engineering, or a related field.
• Hands-on PyTorch experience with large-scale parallelism stacks on multi-node jobs.
• Fluency in Python and C++/CUDA with the ability to predict where a kernel will stall from memory access pattern before profiling.
• Experience profiling live training runs with Nsight Systems or the PyTorch profiler and translating traces into quantifiable step-time or MFU improvements.
• Expertise in at least one of low-precision numerics, kernel authoring, or large-run fault diagnosis.

Responsibilities

• Distributed Training Throughput: Own step time and model FLOPs utilization for multi-node video world model training, choosing tensor and expert parallelism.
• Precision & Numerical Stability: Take BF16, FP8, and NVFP4 recipes from running to converging on Blackwell, addressing scaling-factor and accumulation bugs in video tokenizer and VAE layers.
• Kernels & Compilation: Write and tune CUDA and Triton kernels to enable FlashAttention-4, FlexAttention, and torch.compile for long sequences.
• Communication & Overlap: Tune NCCL collectives and compute/communication overlap across NVLink domains and the fabric.
• Fault Diagnostics & Recovery: Build detection for silent data corruption, stuck kernels, and asynchronous checkpointing to minimize interruption cost.

Skills

Python

C++/CUDA

PyTorch

Profiling

Education

Bachelor's degree or equivalent hands-on experience in Computer Science, Computer Engineering, or a related technical field

Tools

Nsight Systems

PyTorch Profiler

Megatron-Core

DeepSpeed

TorchTitan

CUDA

Triton

Job description

Veeda AI in Toronto is seeking a Member of Technical Staff - ML Performance to own step time and model throughput for multi-node video world model training, optimize tensor usage, and implement advanced parallelism.

You will write and tune CUDA and Triton kernels, improve precision and numerical stability, and build fault diagnostics and elastic checkpointing to minimize downtime during interruptions.