Cluster Site Reliability Engineer

iFrame

Toronto, Montreal , Vancouver

On-site

CAD 299,000 - 484,000

Full time

5 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

iFrame in Canada is seeking a Cluster Site Reliability Engineer to own the physical reality of the platform across seven regions. You will bring up new GPU racks, validate InfiniBand fabric end-to-end, and keep the cluster running at the SLA. This is a hands-on role and you will see the hardware.

Location is on-site with multiple region options. The role covers capacity planning, incident response, and creating a robust bring-up runbook for future hires, while collaborating across regions and

Qualifications

  • Five-plus years operating large compute clusters.
  • Deep InfiniBand and Ethernet RoCE experience with fabric tuning.
  • Strong Linux fundamentals: kernel params, NUMA topology, cgroups.
  • Comfort writing Go or Python for tooling.
  • Calm under load; named on a $50M-ARR account when issues arise.

Responsibilities

  • Bring up new GPU racks: cabling, ToR, validation, MFU tests.
  • Drive InfiniBand fabric to spec and reduce errors to zero.
  • Forecast demand and model power/thermal headroom across regions.
  • Own regional incident response with defined SLA targets.
  • Build/run the bring-up runbook for future hires.
  • Manage the global pager and coordinate regional duties.

Skills

Linux
Go
Python
Kubernetes
InfiniBand
NCCL/RCCL
Prometheus
Grafana

Tools

Terraform
NVIDIA Base Command Manager
Tinkerbell
MAAS
Razor

Job description

Cluster & SRE
Cluster Site Reliability Engineer

Own the physical reality of the platform in one of our seven regions. You bring up new GPU racks, validate InfiniBand fabric end-to-end, and keep the cluster running at the SLA. This is a hands‑on role; you will see the hardware.

Team Cluster & SRE

Location: On‑site

  • Cologix region of choice (Toronto, Montréal, Columbus, Vancouver, Ashburn)

Type: Full‑time

Stack
  • Linux
  • InfiniBand (NDR / XDR)
  • NCCL / RCCL
  • Kubernetes (host-level)
  • Terraform
  • Prometheus / Grafana
  • Go or Python
About The Team

Cluster SRE is six engineers across the seven regions. Each region has a primary and a secondary; you will be one of those for your region. The team coordinates daily, deploys weekly, and rotates a global pager.

Reports to the head of cluster engineering. Primary on a single region; rotates secondary for one neighboring region.

The role
What you'll do
  • 01 Bring up new B200 / B300 / MI300X racks: cabling, ToR config, NCCL/RCCL all-reduce validation, MFU baseline tests.
  • 02 Drive InfiniBand fabric to spec — NDR / XDR depending on the rack — and chase residual bit-error budget down to zero.
  • 03 Run capacity planning across seven regions: forecast demand, model power and thermal headroom, work with procurement on lead times.
  • 04 Own the regional incident response. P1 incidents page within fifteen minutes; resolution target is four hours.
  • 05 Build and maintain the bring‑up runbook so the second hire after you can do their first rack solo.
  • 06 Carry the global pager about one week per six, alongside runtime and customer engineering.
What we're looking for
  • Five-plus years operating large compute clusters — supercomputing centers, hyperscaler infra, or HPC at a national lab count.
  • Deep InfiniBand and Ethernet RoCE experience: subnet manager tuning, fabric debugging, lossless networking.
  • Strong Linux fundamentals: kernel parameters, NUMA topology, kernel cgroup limits.
  • Comfort writing Go or Python for tooling. We are not strict about which.
  • Calm under load. You will be the named person on a $50M-ARR account when something goes wrong.
Bonus
Nice to have, not required
  • DGX H100 / H200 / B200 bring‑up history.
  • Experience with NVIDIA Bright / Base Command Manager.
  • Bare‑metal provisioning systems: Tinkerbell, MAAS, Razor, or in‑house equivalents.
  • Procurement / vendor management experience.
Compensation

In writing, like everything else

Base $210,000 – $340,000 USD (US Cologix regions) / equivalent in CA.

Equity Meaningful early‑stage equity, refreshed on tenure milestones.

Notes

On‑site pay differential at Cologix regions outside SF / NYC / Bay Area is +5–10% to compensate for travel.

Equal opportunity

We hire on the work. Race, gender, age, nationality, religion, sexual orientation, disability, and veteran status do not factor into our decisions. We sponsor visas for senior roles in the US, UK, and EU — bring it up on the manager call.

Need an accommodation for the interview process? Mention it in your application or write careers@iframe.ai.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Cluster Site Reliability Engineer
Cluster Site Reliability Engineer

iFrame Corporation • Canada

On-site
CAD 294,000 - 476,000
Equity
Senior Cluster SRE: GPU Racks, InfiniBand & On‑Site
Senior Cluster SRE: GPU Racks, InfiniBand & On‑Site

iFrame • Toronto, Montreal , Vancouver

On-site
CAD 299,000 - 484,000
Software Engineer, ML Workload Platform
Software Engineer, ML Workload Platform

AItoolnavio • Toronto

Hybrid
CAD 120,000 - 180,000
Weekly lunch stipend
Health and dental benefits
RRSP matching
+5
Software Engineer, ML Workload Platform
Software Engineer, ML Workload Platform

Cohere • Toronto, Montreal

Remote
CAD 90,000 - 140,000
Lunch stipend
Health & dental benefits
RRSP matching
+4
Engineering Manager, GPU Infrastructure
Engineering Manager, GPU Infrastructure

Precision Labs • Toronto

Hybrid
CAD 120,000 - 180,000
Lunch stipend
Health & dental benefits
RRSP matching / pension
+5
Site Reliability Engineer, AI/ML Infrastructure
Site Reliability Engineer, AI/ML Infrastructure

Boson AI • Toronto

On-site
CAD 100,000 - 130,000
Cloud - Bare Metal Infrastructure Engineer
Cloud - Bare Metal Infrastructure Engineer

Jobgether • Canada

On-site
CAD 321,000 - 405,000
Equity participation
Four weeks paid vacation
Parental leave (14 weeks)
Forward Deployed Engineer, Infrastructure Specialist (North America)
Forward Deployed Engineer, Infrastructure Specialist (North America)

FDE Times • Toronto

Hybrid
CAD 110,000 - 170,000
Health and dental benefits
6 weeks vacation
Remote-friendly with global offices
Cloud - Bare Metal Infrastructure Engineer
Cloud - Bare Metal Infrastructure Engineer

Render • Canada

Remote
CAD 140,000 - 190,000
4 weeks of paid vacation
14 weeks of parental leave
Long‑term disability, life insurance,
+6
Software Engineer - GPU Networking & Distributed Systems
Software Engineer - GPU Networking & Distributed Systems

Baseten • Montreal

On-site
CAD 230,000 - 461,000
Competitive compensation
Equity
Flexible PTO
+2