Principal Engineer Software

Palo Alto Networks, Inc.; Santa Clara (CA); On-site; USD 147,000 - 237,500; Full time

Benefits offered by this job

Comprehensive health insurance

401(k) retirement plan

Competitive salary

Job summary

Palo Alto Networks, Inc. is seeking a Senior Data Center & OpenShift Operations Engineer in Santa Clara, CA. The role involves ensuring high availability and reliability of data center systems while managing OpenShift clusters.

The ideal candidate will have a Bachelor's degree in Computer Science or IT, at least 5 years of experience with Red Hat OpenShift, and strong skills in Python, Bash, and automation tools like Ansible. This position offers a competitive salary and a comprehensive benefits package.

Qualifications

• 5+ years experience in Red Hat OpenShift in a production environment.
• Deep experience in racking and cabling high-density GPU systems.
• Strong Python and Bash scripting skills for health checks and API integrations.

Responsibilities

• Monitor and maintain data center systems for high availability.
• Implement and manage OpenShift 4.x clusters to ensure uptime.
• Design and execute automated disaster recovery strategies.

Skills

Red Hat OpenShift (OCP)

Ansible

Python

Bash

CoreOS

RHEL administration

GPU systems

Networking (BGP, VLAN)

Education

Bachelor’s degree in Computer Science or IT

Tools

vSphere

KVM

OpenShift Data Foundation (ODF)

Ceph

DCIM tools (Netbox)

Monitoring stacks (ELK/Loki)

Job description

Senior Data Center & OpenShift Operations Engineer

Position Overview: The Senior Data Center Operations Engineer is responsible for the core of our high‑availability infrastructure. This role bridges the gap between physical hardware and the Red Hold OpenShift Container Platform (OCP). Your mission is to ensure 99.99% availability by architecting resilient physical layouts and automating the deployment, scaling, and self‑healing of our production clusters.

Key Responsibilities

• High‑Availability (HA) Infrastructure: Monitor and maintain data center systems with a focus on Zero Single Point of Failure (ZSPoF) architecture for OpenShift control planes and worker nodes.
• Cluster Reliability Engineering: Implement and manage OpenShift 4.x clusters across multiple power and cooling zones to ensure 99.99% uptime.
• Disaster Recovery & Business Continuity: Design, test, and execute automated failover strategies and backup/restore procedures using OADP (Velero) and Red Hold ACM.
• Automated Maintenance: Perform routine maintenance and upgrades using GitOps (ArgoCD) and the Machine Config Operator to ensure zero‑downtime node evacuations and patching.
• Complex Troubleshooting: Resolve deep‑stack hardware and software issues, from faulty GPU firmware to OpenShift SDN (OVN‑Kubernetes) network latencies.
• Vendor & Lifecycle Management: Coordinate with vendors for specialized hardware (e.g., NVIDIA, Dell, Cisco) while maintaining strict security and firmware compliance.
• Efficiency & Capacity Architecture: Optimize rack density for high‑performance GPU clusters while managing thermal loads and power distribution (PDU) to prevent circuit‑trip outages.
• Observability Implementation: Maintain accurate documentation and integrate hardware health metrics (IPMI/SNMP) into Prometheus/Grafana for proactive alerting.
• Physical Deployment: Rack and stack high‑density GPU servers, ensuring redundant power‑pathing and high‑speed (100G/200G) InfiniBand or Ethernet cabling.
• Hardware Lifecycle: Perform precision physical installation and replacement of critical components (CPUs, GPUs, NVMe storage) in a live production environment without impacting cluster quorum.

Qualifications

• Education: Bachelor’s degree in Computer Science, IT, or equivalent experience.
• Platform Expertise: 5+ years of experience specifically operating Red Hold OpenShift (OCP) in a production environment.
• Hardware Fluency: Deep experience racking/stacking and cabling high‑density GPU systems (e.g., NVIDIA DGX or similar) and specialized AI/ML hardware.
• Infrastructure as Code (IaC): Advanced proficiency in Ansible or Pulumi for automating bare‑metal provisioning and cluster configuration.
• Scripting: Strong Python and Bash skills for developing custom health‑check scripts and API integrations.
• Linux Mastery: Expert‑level CoreOS and RHEL administration, including kernel tuning and systemd management.
• Networking: Solid understanding of BGP, VLAN tagging, LACP, and Load Balancing (F5/NGINX) essential for cluster ingress.
• Virtualization & Storage: Experience with vSphere or KVM, and persistent storage solutions like OpenShift Data Foundation (ODF) or Ceph.
• Tooling: Familiarity with DCIM tools (Netbox) and monitoring stacks (ELK/Loki, etc.).

Physical Requirements

• Lifting: Ability to lift and move equipment up to 50 pounds.
• Environment: Comfortable working in high‑decibel, climate‑controlled data center aisles.
• Dexterity: Capable of standing, walking, and performing precision cabling in tight rack spaces for extended periods.
• Travel: May require occasional travel to remote data center sites or edge locations.

Benefits

• Competitive salary commensurate with high‑availability expertise.
• Comprehensive health, dental, and vision insurance.
• 401(k) retirement plan with company match.

Compensation Disclosure

The compensation offered for this position will depend on qualifications, experience, and work location. For candidates who receive an offer at the posted level, the starting base salary is expected to be in the range $147,000.00 – $237,500.00 per year. The offered compensation may also include restricted stock units and a bonus.

Equal Employment Opportunity

We are committed to providing reasonable accommodations for all qualified individuals with a disability. If you require assistance or accommodation due to a disability or special need, please contact us at [e-mail].

Palo Alto Networks is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.

All your information will be kept confidential according to EEO guidelines.