ML Ops Engineer - Reproducible AI Pipelines

Veeda AI; Toronto; On-site; CAD 120,000 - 150,000; Full time

Job summary

Veeda AI in Toronto is seeking a hands-on ML/software engineer to own end-to-end tooling for lifecycle management, experiment tracking, and checkpoint lineage on multi-node training jobs.

You will ensure reproducible pipelines, robust CI/CD practices (GitHub Actions, Buildkite), and responsive incident handling, with a strong Python background and proven software craftsmanship. Candidates should have experience shipping internal tooling that others adopt.

Qualifications

• Bachelor's degree or equivalent hands-on experience in Computer Science, Engineering, or a related technical field.
• Strong Python and software engineering skills and real CI/CD experience (e.g., GitHub Actions, Buildkite), and you have shipped internal tooling that other engineers chose to keep using.
• You have operated multi-node training jobs, carried the pager for them, and decided from telemetry whether to kill, requeue, or let a degraded run ride.
• You have built reproducible pipelines end to end, and can say precisely which parts of a training run are bit-reproducible, which are not, and why.
• You are rigorous about evaluation methodology, from seeds and sample sizes to confidence intervals, and can tell a genuine regression from a flaky harness.

Responsibilities

• Run Lifecycle & Launch Tooling: Define, launch, resume, and kill runs from typed configs to pinned container digests.
• Experiment Tracking & Provenance: Bind every checkpoint to code commit, config hash, dataset version, and container digest in ML tools.
• Checkpoint Registry & Lineage: Own retention, GC policy, and format conversions for artifacts used by simulation/robotics.
• Evaluation in CI: Gate checkpoints on seeded rollout and policy-success suites in CI workflows.
• Goodput & Incident Response: Be on call for live runs and report goodput against allocated GPU-hours.

Skills

Python

CI/CD

Multi-node training

Reproducible pipelines

Experiment tracking

Docker / container tooling

Telemetry / monitoring

Education

Bachelor's degree or equivalent hands-on experience in Computer Science, Engineering, or related field

Tools

GitHub Actions

Buildkite

Weights & Biases

MLflow

Argo Workflows

Flyte

Ray

Prometheus

Grafana

Job description

Veeda AI in Toronto is seeking a hands-on ML/software engineer to own end-to-end tooling for lifecycle management, experiment tracking, and checkpoint lineage on multi-node training jobs.

You will ensure reproducible pipelines, robust CI/CD practices (GitHub Actions, Buildkite), and responsive incident handling, with a strong Python background and proven software craftsmanship. Candidates should have experience shipping internal tooling that others adopt.