Member of Technical Staff - AI Infrastructure

Veeda AI; Toronto; On-site; CAD 150,000 - 230,000; Full time

Job summary

Veeda AI is building next-generation multimodal foundation models for Physical AI in a tight-knit team in Toronto. You will design and operate Slurm/Kubernetes GPU clusters, ensuring performance and reliability across compute, storage, and networking, while implementing secure access, observability, and cost-aware planning.

As a senior infra engineer, you’ll forecast resource needs with Model and Data teams, drive PoCs with vendors, and own CI/CD, production observability, and on-call support in

Qualifications

• Bachelor’s degree in Computer Science, Computer Engineering, or equivalent hands-on HPC or infrastructure experience.

Responsibilities

• Design, deploy, and operate Slurm-on-Kubernetes GPU clusters and manage compute, interconnect, storage, and scheduler resources.

Skills

HPC clusters

Linux networking

Automation

Infrastructure as code

Distributed DL workloads

Security policies

Slurm/Kubernetes

Education

Bachelor's degree in CS/CE or equivalent

Tools

Ansible

Terraform

Helm

Python/Bash scripting

GitLab CI

Docker

Kubernetes operators

Prometheus

Grafana

OpenTelemetry

Slurm

Job description

About Us

Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

About Us

Veeda AI is building the next generation of multimodal foundation world models for Physical AI. We're a small, fast-moving team of engineers and researchers from leading AI labs, tackling some of the most challenging problems at the intersection of AI, robotics, and embodied intelligence. If you're excited about pushing the boundaries of what's possible with Physical AI, you'll have the opportunity to make an outsized impact from day one.

Responsibilities

• GPU Cluster Operations: Design, deploy, and operate Slurm-on-Kubernetes GPU clusters. Own the compute (CPU/GPU device health), interconnect (nvlink, Infiniband, RoCE), storage (linux file systems, Lustre), and scheduler (topology aware scheduling, QoS, partitions, fair-share) resources.
• Access Control and Security: Design, deploy, and operate identity provisioning and access control for our source code, cluster, knowledge bases, data storages, and dev tools.
• Network and Data: Design and maintain distributed data transmission and caching services that provide just-in-time delivery of data and checkpoints to the clusters in a timely and cost-effective manner. Collaborate with internal stakeholders and external vendors to find the optimal path between data and compute.
• Observability & Hardware Health: Build the telemetry pipeline (Prometheus, Grafana, DCGM, BMC/Redfish) that catches Xid and ECC errors, thermal throttling, and link flaps, and work with cloud vendors to diagnose and resolve issues in timely manner.
• Resource Forecasting: Work with Model and Data teams to determine resource requirements and delivery cadence for the next month/quarter/year. Lead PoC rounds with vendors for compute and data resources.
• Infra-OPEX: Prepare operational expenses forecast and reports. Provide expert opinion on key infra spending decisions.
• End-to-End Ownership: Lead projects through the complete software lifecycle, including technical specs, implementation, CI/CD, on-call support, and production observability.

Requirements

• Bachelor's degree in Computer Science, Computer Engineering, or equivalent hands-on experience in high-performance computing (HPC) or infrastructure engineering.
• Strong troubleshooting skills below the framework layer: low-level Linux networking, kernel and PCIe/NUMA tuning, hardware diagnostics, and network or distributed storage (NFS, NVMe-oF, Lustre, Ceph).
• Proficient in automation and infrastructure-as-code tools (e.g., Ansible, Terraform, Helm, Python/Bash scripting), and treats cluster configuration as reviewed, version-controlled code.
• Experience supporting distributed deep learning workloads (PyTorch/NCCL, Ray, DeepSpeed) closely enough to tell an infrastructure fault from a model bug, and able to prove which with a controlled benchmark.
• One of the following:
  • Deep hands-on expertise administering Linux-based HPC clusters running Slurm or Kubernetes (job scheduling, cgroups, fair-share policies, topology configuration, GPU device plugins).
  • Experience working with cloud providers on workload profiling, Proof of Concepts (PoC), resource cost strategies.
  • Experience managing security policies. Hands‑on‑experience with building security layers from overall strategy to IaC. Educating fast moving Dev teams of security practices.

Nice to Have

• Experience managing high-density GPU infrastructure (NVIDIA H100/H200, B200, and GB200 NVL72 systems, DGX/HGX architectures, liquid‑cooled racks).
• Built container and image supply chains for HPC workloads (Enroot, Pyxis, Docker, custom Kubernetes operators).
• Experience running hybrid capacity, combining owned hardware with cloud or neocloud burst under a single scheduler.
• Experience running secure multi‑tenant research environments with SSO, per‑team quota, and interactive access that stays fast.
• Fluency with Prometheus, Grafana, and OpenTelemetry, and instrument workloads before its first outage.
• Contributed to Slurm plugins, Kubernetes operators, or open‑source cluster and observability tooling.
• Experience with GitLab, especially GitLab CI, for managing infrastructure‑as‑code and automation pipelines.
• Experience benchmarking fabric, storage, or scheduler performance and publishing results internally or externally to settle a design or procurement decision.