Principal Production Engineer – Automation & Global Reliability

Zscaler; United States; Hybrid; USD 165,000 - 235,000; Full time

Benefits offered by this job

Health plans

Retirement options

Education reimbursement

In-office perks

Job summary

Zscaler is seeking a Principal Production Engineer to drive highly available, scalable infrastructure across cloud and bare-metal environments. You will lead an automation-first approach, maturing observability and architectural standards to reduce MTTR and scale a global platform handling billions of transactions daily.

The role reports to Production Engineering within Cloud Infrastructure & Operations and offers hybrid work options (3 days/week in San Jose, CA or remote).

Qualifications

• 10+ years of experience in reliability, scalability, and availability for large-scale production services
• Deep programming experience (Python, Go, or C/C++)
• Strong Linux/RHEL, networking, and distributed systems background
• Proven incident management with 24/7 on-call rotation
• Familiarity with ITIL frameworks and incident data usage to drive maturity

Responsibilities

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
• Drive an automation-first culture by writing code to reduce toil and build self-healing systems
• Implement and maintain observability stack (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and manage error budgets
• Act as lead Incident Commander (TDO on-call), develop response playbooks, and conduct post-incident analyses
• Partner with engineering teams to perform operability reviews

Skills

Python

Go

C/C++

Linux

Incident management

ITIL

Observability

Tools

Prometheus

Grafana

OpenTelemetry

Ansible

Terraform

Helm

Job description

Zscaler is seeking a Principal Production Engineer to drive highly available, scalable infrastructure across cloud and bare-metal environments. You will lead an automation-first approach, maturing observability and architectural standards to reduce MTTR and scale a global platform handling billions of transactions daily.

The role reports to Production Engineering within Cloud Infrastructure & Operations and offers hybrid work options (3 days/week in San Jose, CA or remote).