Production Reliability Engineer, AI Platform

Equinix

Toronto

On-site

CAD 102,000 - 152,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Employee Assistance Program

Job summary

Equinix's team in Toronto is seeking a Reliability Engineer to maintain GenAI products and platforms, define SLOs, and lead incident response in production. You will implement monitoring, automation, and resilient deployments with Kubernetes, CI/CD, and IaC practices, collaborating with AI engineers, product managers and security partners.

You will run load tests, improve runbooks, and ensure services meet performance targets while scaling with demand, helping to deliver secure, observable and

Qualifications

  • 1–4 years in site reliability, DevOps, platform or software engineering with production systems.
  • Bachelor's degree in computer science, engineering or related field, or equivalent experience.
  • Proficiency in Python or Go with solid software engineering habits.
  • Experience with Linux, containers, Kubernetes and a major cloud platform.
  • Experience with CI/CD and infrastructure as code (Terraform).
  • Monitoring, logging, tracing and alerting knowledge.
  • Understanding of distributed systems failure and resilience patterns.
  • On-call experience and willingness to join on-call rotation.
  • Familiarity with LLM applications and model APIs.

Responsibilities

  • Write and automate for GenAI reliability; implement retries, timeouts, and circuit breakers.
  • Run load tests and game days to expose weak points and fix them.
  • Harden Kubernetes services with health checks, autoscaling and multi‑zone deployment.
  • Plan capacity ahead of launches, including GPU and inference capacity.
  • Maintain backup, restore and disaster-recovery plans and test recovery regularly.
  • Automate repetitive tasks to reduce manual work.

Skills

Python/Go
Linux
Kubernetes
CI/CD
On-call
Distributed systems

Education

Bachelor's degree in computer science or engineering

Tools

Terraform
OpenTelemetry
Prometheus
Grafana
GitOps (Argo CD)
GCP

Job description

Who are we?

Equinix is the world’s digital infrastructure company®, shortening the path to connectivity to enable the innovations that enrich our work, life and planet.

A place where tech thinkers and future builders turn bold ideas into breakthrough experiences, we welcome your unique perspective.Help us challenge assumptions, uncover bias, and remove barriers—because progress starts with fresh ideas. You’ll find belonging, purpose, and a team that welcomes you—because when you feel valued, you’re empowered to do your best work.

Job Summary

Keeps Equinix's GenAI products and platforms stable and reliable, and makes them more resilient over time. Writes the code and automation that prevents failures, rather than only responding to them. Defines service-level objectives, builds monitoring and alerting, hardens services against dependency and capacity failures, and tests failure and recovery before they happen in production. Leads incident response when issues occur and turns each incident into a lasting fix. Works in a team with senior engineers and with product, platform and security partners to keep AI products, agents, connectors and self-hosted models robust, secure, observable and performant.

Knowledge and Application

Applies solid knowledge of software engineering and site reliability engineering (SRE) practice: service-level objectives (SLOs), error budgets, monitoring, alerting, incident response, CI/CD and infrastructure as code. Understands how distributed systems fail and how to contain those failures (timeouts, retries, circuit breakers, graceful degradation). Understands how GenAI products behave in production, including model latency, rate limits, quotas, streaming and token cost. Follows established standards and suggests improvements.

Complexity and Problem Solving

Works on problems of defined scope where analysis requires evaluation of identifiable factors. Diagnoses failures across application, model, data, pipeline and infrastructure layers, and fixes the underlying cause rather than the symptom. Applies existing runbooks and approaches, and improves them. Escalates unusual or high-impact issues. Work is reviewed at defined checkpoints.

Collaboration and Interaction

Works with AI engineers, product managers, platform and security teams. Reviews designs and releases for reliability risk before they ship. Explains status, incidents and results clearly to partners who are not familiar with the subject. Asks for guidance early and shares knowledge with the team.

Responsibilities
Reliability and resilience engineering (primary focus)
  • Write code and automation that makes GenAI products and platforms more resilient: retries, timeouts, circuit breakers, rate limiting, graceful degradation and fallback between models and providers
  • Run load tests, failure-injection tests and game days to find weak points before users do, and fix what they reveal
  • Harden services running on Kubernetes: health checks, autoscaling, disruption budgets, resource limits and multi-zone deployment
  • Plan capacity ahead of launches and growth, including model-provider quotas, GPU and inference capacity, and database headroom
  • Maintain backup, restore and disaster-recovery plans, and test recovery on a regular schedule
  • Automate repeated operational tasks, and reduce manual work each quarter
Monitoring and service levels
  • Define and track service-level indicators and objectives with product owners: availability, latency, error rate, token cost and quality scores
  • Instrument services with logs, metrics and traces (e.g., OpenTelemetry), from the user request through agent, tool and model calls
  • Build and tune dashboards and alerts for user-facing failures, and remove alerts that are noisy or not actionable
Release and deployment
  • Support release readiness for GenAI products and platforms: checks for monitoring, alerts, runbooks, capacity, security review and rollback plan
  • Deploy through CI/CD and GitOps pipelines, using containers and infrastructure as code, with canary rollouts and fast rollback
Incident response
  • Take part in the on-call rotation, respond to incidents, and write blameless post-incident reviews
  • Track post-incident actions to completion, so the same failure does not happen twice
  • Write and update runbooks, and turn recurring incidents into automation or fixes
GenAI and self-hosted models
  • Support the reliability of self-hosted open-weight and small language models: serving, GPU usage, scaling, rollout and rollback
  • Monitor model and provider latency, errors, rate limits and cost, and flag capacity and cost risks
  • Apply security, patching, secrets and data-boundary requirements to every deployment
Qualifications
Required
  • 1–4 years in site reliability, DevOps, platform or software engineering, with hands‑on work on production systems
  • Bachelor's degree in computer science, engineering or a related field, or equivalent experience
  • Proficiency in Python or Go, with solid software engineering habits: version control, testing, code review
  • Experience with Linux, containers, Kubernetes and a major cloud platform (Google Cloud preferred)
  • Experience with CI/CD and infrastructure as code (e.g., Terraform)
  • Working knowledge of monitoring, logging, tracing and alerting
  • Understanding of how distributed systems fail and of common resilience patterns
  • Experience with on-call or incident response, and willingness to join an on-call rotation
  • Familiarity with LLM applications (model APIs, retrieval, agents, tool calls)
  • Clear written and verbal communication
Nice to have
  • Experience running GenAI or ML systems in production, including managed model platforms (e.g., Vertex AI) or model serving (e.g., vLLM)
  • Experience with observability tools (e.g., OpenTelemetry, Prometheus, Grafana) and incident tools (e.g., PagerDuty)
  • Experience with GitOps (e.g., Argo CD), Helm and canary deployments
  • Experience with load testing or chaos engineering
  • Experience with secrets management (e.g., Vault) and security scanning
  • Experience building automation or AI agents that reduce operational work
What success looks like
  • GenAI products and platforms meet their service-level objectives, and releases ship without avoidable incidents
  • Alerts catch failures before users report them, and each incident ends with a documented fix
  • The same incident does not happen twice
  • Load tests and recovery tests run on schedule, and launches scale without capacity issues
  • Manual operational work goes down each quarter
  • The engineer takes on larger, less defined work each quarter with decreasing guidance

The targeted pay range for this position in the following location is / locations are:

Canada - Toronto Office TRO : 102,000 - 152,000 CAD / Annual

United States - Dallas Infomart Office DAI : 138,000 - 208,000 USD / Annual

Our pay ranges reflect the minimum and maximum target for new hire pay for the full-time position determined by role, level, and location.The pay range shown is based on our compensation structure in place at the time of posting and may be updated periodically based on business needs. Individual pay is based on additional factors including job-related skills, experience, and relevant education and/or training.

The targeted pay range listed reflects the base pay only and does not include bonus, equity, or benefits. Employees are eligible for bonus, and equity may be offered depending on the position.

Equinix Benefits

As an employee, you become important to Equinix’s success. We ensure all your benefits are in line with our core values: competitive, inclusive, sustainable, connected and efficient. We keep them competitive within the current marketplace to ensure we’re providing you with the best package possible. So, wherever you are in your career and life, you’ll be able to enhance your experience and bring your whole self to work.

Employee Assistance Program

An Employee Assistance program is available to all employees.

US Benefits: - Insurance: You may enroll in health, life, disability and voluntary plans that are designed for you and your eligible family members. - Retirement: You and Equinix may contribute to a retirement plan to help you plan for your financial future. - Paid Time Off (PTO) and Paid Holidays: You will receive an accrued amount of PTO each pay period along with various paid holidays for you to rest and recharge. Eligibility requirements apply to some benefits. Benefits are subject to change and may be subject to specific plan or program terms.Canada Core Benefits: - Insurance: You may enroll in healthcare coverage that is designed to complement the provincial healthcare system, along with life, disability and optional benefit plans that are designed for you and your eligible family members. - Retirement: You may also enroll in Equinix-sponsored retirement or savings plans: Defined Contribution Pension Plan (DCPP), Group Retirement Savings Plan (RRSP) and Tax-Free Savings Plan (TSFA). - Vacation and Paid Holidays: Equinix offers both vacation and personal time, along with various paid holidays for you to rest and recharge. Eligibility requirements apply to some benefits. Benefits are subject to specific plan or program terms, and to change at Equinix discretion.

Equinix is committed to ensuring that our employment process is open to all individuals, including those with a disability. If you are a qualified candidate and need assistance or an accommodation, please let us know by completing this form.

Equinix is an Equal Employment Opportunity and, in the U.S., an affirmative Action employer. All qualified applicants will receive consideration for employment without regard to unlawful consideration of race, color, religion, creed, national or ethnic origin, ancestry, place of birth, citizenship, sex, pregnancy / childbirth or related medical conditions, sexual orientation, gender identity or expression, marital or domestic partnership status, age, veteran or military status, physical or mental disability, medical condition, genetic information, political / organizational affiliation, status as a victim or family member of a victim of crime or abuse, or any other status protected by applicable law.

We use artificial intelligence in our hiring process. Learn more here.

This posting is a new position within our organization.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI Engineer. Innovation and R&D
Senior AI Engineer. Innovation and R&D

Equinix • Toronto

On-site
CAD 142,000 - 212,000
Senior AI Engineer. Innovation and R&D
Senior AI Engineer. Innovation and R&D

Equinix • Toronto

On-site
CAD 142,000 - 212,000
Equinix Benefits
Employee Assistance Program
Senior Director AI, Chief Revenue Organization
Senior Director AI, Chief Revenue Organization

Equinix • Toronto

On-site
CAD 182,000 - 272,000
Employee Assistance Program
Canada Benefits (RRSP / TSFA)
Vacation and Paid Holidays
AI/Machine Learning Experience Engineer
AI/Machine Learning Experience Engineer

Equinix • Toronto

On-site
CAD 102,000 - 152,000
Principal, AI Technical Expert
Principal, AI Technical Expert

Equinix • Toronto

On-site
CAD 142,000 - 212,000
Principal Software Engineer, AI Infrastructure
Principal Software Engineer, AI Infrastructure

Equinix • Vancouver

On-site
CAD 142,000 - 212,000
Equinix Benefits
Employee Assistance Program
Senior Manager, Digital Strategy & Operations
Senior Manager, Digital Strategy & Operations

Equinix • Toronto

On-site
CAD 115,000 - 165,000
Employee Assistance Program
Vacation and paid holidays
Competitive benefits
Senior Director, Process Excellence
Senior Director, Process Excellence

Equinix • Toronto

On-site
CAD 206,000 - 308,000
Healthcare coverage
Retirement plan (RRSP/TFSA)
Vacation & holidays
Process Excellence Senior Manager
Process Excellence Senior Manager

Equinix • Toronto

On-site
CAD 139,000 - 209,000
Principal Product Manager, AI Infrastructure
Principal Product Manager, AI Infrastructure

Equinix • Toronto

On-site
CAD 170,000 - 254,000
Employee Assistance Program
Canada Benefits overview
Retirement and RRSP/TSFA options