Principal Production Engineer

Zscaler; United States; Hybrid; USD 165,000 - 235,000; Full time

Benefits offered by this job

Health plans

Retirement options

Education reimbursement

In-office perks

Job summary

Zscaler is seeking a Principal Production Engineer to drive highly available, scalable infrastructure across cloud and bare-metal environments. You will lead an automation-first approach, maturing observability and architectural standards to reduce MTTR and scale a global platform handling billions of transactions daily.

The role reports to Production Engineering within Cloud Infrastructure & Operations and offers hybrid work options (3 days/week in San Jose, CA or remote).

Qualifications

• 10+ years of experience in reliability, scalability, and availability for large-scale production services
• Deep programming experience (Python, Go, or C/C++)
• Strong Linux/RHEL, networking, and distributed systems background
• Proven incident management with 24/7 on-call rotation
• Familiarity with ITIL frameworks and incident data usage to drive maturity

Responsibilities

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
• Drive an automation-first culture by writing code to reduce toil and build self-healing systems
• Implement and maintain observability stack (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and manage error budgets
• Act as lead Incident Commander (TDO on-call), develop response playbooks, and conduct post-incident analyses
• Partner with engineering teams to perform operability reviews

Skills

Python

Go

C/C++

Linux

Incident management

ITIL

Observability

Tools

Prometheus

Grafana

OpenTelemetry

Ansible

Terraform

Helm

Job description

Zscaler (NASDAQ: ZS) accelerates digital transformation so customers can be more agile, efficient, resilient, and secure. The Zscaler Zero Trust Exchange platform protects thousands of customers from cyberattacks and data loss by securely connecting users, devices, and applications in any location. Distributed across 200+ public data centers globally and thousands of private sites at the edge, the SASE-based Zero Trust Exchange is the world’s largest in-line cloud security platform.

We believe the future of work is Human + AI and are building an AI-native enterprise where human potential is amplified by machine intelligence to solve the world’s hardest security challenges. Driven by deep customer obsession, we are committed to the mission, outcome, and to each other. We bring these commitments to life through three core behaviors: ownership and collaboration, trust through outcomes and impact, and a challenge culture with ongoing feedback. Ready to make an impact at the company pioneering security transformation in the AI era? Join us at Zscaler.

Role

We are looking for a Principal Production Engineer to join our team. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department. Join Zscaler to be a force multiplier for the reliability of a global platform processing 200+ billion transactions daily across tens of millions of enterprise users.

In this role, you will provide the technical vision and hands‑on execution to drive an "automation-first" culture across the company. By maturing our observability and architectural standards, you will directly reduce our Mean Time to Mitigate (MTTM) and shape the scalability of our globally distributed, multi‑cloud infrastructure.

What you’ll do (Role Expectations)

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare‑metal environments
• Drive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self‑healing systems
• Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
• Act as a lead Incident Commander (TDO on‑call), develop response playbooks, and conduct deep‑dive post‑incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

Who You Are (Success Profile)

• You act like an owner with a bias for action and integrity.
• You are a pragmatic builder obsessed with creating, iterating, and shipping.
• You champion simplicity by distilling complex problems into clear, actionable plans.
• You are data‑driven, valuing evidence over assumptions.
• You think at scale, building solutions and processes built to last a high‑growth global organization.

What We’re Looking for (Minimum Qualifications)

• Foundational understanding of AI/ML technologies and experience leveraging, securing, or positioning AI‑driven solutions to optimize outcomes within your functional domain
• 10+ years of experience managing reliability, scalability, and availability for large‑scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
• Experience in high‑stakes incident management and participation in a 24/7 on‑call rotation
• Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

What Will Make You Stand Out (Preferred Qualifications)

• Proven ability to leverage AI technologies and workflows to drive measurable operational efficiency
• Extensive experience with public cloud (AWS, Azure, GCP) and Infrastructure‑as‑Code (Ansible, Terraform, Helm, Temporal)
• Experience with chaos engineering and disaster recovery planning at scale
• Expertise in global routing (BGP) and traffic tunneling (GRE, IPSec) with a deep understanding of L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internals

#LI-Hybrid #LI-RT101

Zscaler’s salary ranges are benchmarked and are determined by role and level. The range displayed on each job posting reflects the minimum and maximum target for new hire salaries for the position across all US locations and could be higher or lower based on a multitude of factors, including job-related skills, experience, and relevant education or training.

The base salary range listed for this full‑time position excludes commission/ bonus/ equity (if applicable) + benefits.

Base Pay Range

$164,500—$235,000 USD

At Zscaler, we are committed to building a team that reflects the communities we serve and the customers we work with. We foster an inclusive environment that values all backgrounds and perspectives, emphasizing collaboration and belonging. Join us in our mission to make doing business seamless and secure.

Our Benefits program is one of the most important ways we support our employees. Zscaler proudly offers comprehensive and inclusive benefits to meet the diverse needs of our employees and their families throughout their life stages, including:

• Various health plans
• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• Education reimbursement
• In‑office perks, and more!

Learn more about Zscaler's hybrid working model and benefits here.

By applying for this role, you adhere to applicable laws, regulations, and Zscaler policies, including those related to security and privacy standards and guidelines.

Zscaler is committed to providing equal employment opportunities to all individuals. We strive to create a workplace where employees are treated with respect and have the chance to succeed. All qualified applicants will be considered for employment without regard to race, color, religion, sex (including pregnancy or related medical conditions), age, national origin, sexual orientation, gender identity or expression, genetic information, disability status, protected veteran status, or any other characteristic protected by federal, state, or local laws. See more information by clicking on the Know Your Rights: Workplace Discrimination is Illegal link.

Pay Transparency

Zscaler complies with all applicable federal, state, and local pay transparency rules.

Zscaler is committed to providing reasonable support (called accommodations or adjustments) in our recruiting processes for candidates who are differently abled, have long term conditions, mental health conditions or sincerely held religious beliefs, or who are neurodivergent or require pregnancy‑related support.