Principal Production Engineer

Zscaler; San Jose (CA); Hybrid; USD 130,000 - 170,000; Full time

Benefits offered by this job

Time off plans for vacation

Parental leave options

Retirement options

In‑office perks

Job summary

Zscaler is seeking a Principal Production Engineer to enhance the reliability of its platform. This hybrid role allows for remote work while collaborating closely with the Cloud Infrastructure & Operations department in San Jose, CA.

The successful candidate will be responsible for driving an ‘automation-first’ culture, developing scalable solutions, and ensuring the observability of complex infrastructure while managing incident responses efficiently.

Benefits include vacation time, parental leave, and retirement options, all within a culture that values innovation and purpose.

Qualifications

• 10+ years of experience managing reliability for large‑scale production services.
• Expertise in programming, preferably in Python or Go.
• Proficient in incident management and on‑call rotations.

Responsibilities

• Design scalable infrastructure across AWS and GCP.
• Implement and maintain observability tools.
• Drive an automation-first culture by writing code.

Skills

Python

Go

C/C++

Networking protocols

Linux/RHEL systems

Distributed architecture

Tools

Prometheus

Grafana

OpenTelemetry

AWS

GCP

Ansible

Terraform

HAProxy

Job description

About Zscaler

Zscaler accelerates digital transformation to ensure our customers can be more agile, efficient, resilient, and secure. As an AI-forward enterprise, we are constantly pushing the envelope, leveraging the world’s largest security data lake to power our cloud-native Zero Trust Exchange platform. This innovation protects our customers from cyberattacks and data loss by securely connecting users, devices, and applications in any location.

Here, impact in your role matters more than title and trust is built on results. We say, impact over activity. We seek innovators who actively use AI to amplify their impact and who thrive in an environment where we leverage intelligent systems to stay ahead of evolving threats. We believe in transparency and value constructive, honest debate—we’re focused on getting to the best ideas, faster. We build high-performing teams that can make an impact quickly and with high quality. To do this, we are building a culture of execution centered on customer obsession, collaboration, ownership, and accountability.

We value high-impact, high-accountability with a sense of urgency where you’re enabled to do your best work and embrace your potential. If you’re driven by purpose, thrive on solving complex challenges, and want to be part of the team that’s helping to secure the AI age, we invite you to bring your talents to Zscaler and help shape the future of cybersecurity.

Role

We are looking for a Principal Production Engineer to join our team. This role is available as a hybrid opportunity 3 days a week in San Jose, CA or Remote reporting to Production Engineering in the Cloud Infrastructure & Operations department. Join Zscaler to be a force multiplier for the reliability of a global platform processing 200+ billion transactions daily across tens of millions of enterprise users.

In this role, you will provide the technical vision and hands‑on execution to drive an “automation‑first” culture across the company. By maturing our observability and architectural standards, you will directly reduce our Mean Time to Mitigate (MTTM) and shape the scalability of our globally distributed, multi‑cloud infrastructure.

What you’ll do (Role Expectations)

• Design and implement highly available, scalable infrastructure across AWS, GCP, and bare‑metal environments
• Drive an “automation‑first” culture by writing code (Python / Go) to eliminate manual toil and build self‑healing systems
• Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
• Act as a lead Incident Commander (TDO on‑call), develop response playbooks, and conduct deep‑dive post‑incident analyses
• Partner with Engineering and partner teams to conduct operability reviews

Who You Are (Success Profile)

• You act like an owner with a bias for action and integrity.
• You are a pragmatic builder obsessed with creating, iterating, and shipping.
• You champion simplicity by distilling complex problems into clear, actionable plans.
• You are data‑driven, valuing evidence over assumptions.
• You think at scale, building solutions and processes built to last a high‑growth global organization.

What We’re Looking for (Minimum Qualifications)

• 10+ years of experience managing reliability, scalability, and availability for large‑scale production services
• Deep expertise in programming (e.g., Python, Go, or C/C++)
• Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
• Experience in high‑stakes incident management and participation in a 24/7 on‑call rotation
• Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

What Will Make You Stand Out (Preferred Qualifications)

• Extensive experience with public cloud (AWS, Azure, GCP) and Infrastructure‑as‑Code (Ansible, Terraform, Helm, Temporal)
• Experience with chaos engineering and disaster recovery planning at scale
• Expertise in global routing (BGP) and traffic tunneling (GRE, IPSec) with a deep understanding of L7 proxy architectures (HAProxy), DNS at scale, and OS networking stack internals

Benefits

• Time off plans for vacation and sick time
• Parental leave options
• Retirement options
• In‑office perks, and more!

Pay Transparency

Zscaler complies with all applicable federal, state, and local pay transparency rules.

Equal Employment Opportunity

Zscaler is committed to providing equal employment opportunities to all individuals. We strive to create a workplace where employees are treated with respect and have the chance to succeed. All qualified applicants will be considered for employment without regard to race, color, religion, sex (including pregnancy or related medical conditions), age, national origin, sexual orientation, gender identity or expression, genetic information, disability status, protected veteran status, or any other characteristic protected by federal, state, and local laws.