Senior Data Platform Architect — Distributed Systems

Socket.dev; California (MO); On-site; USD 147,000 - 238,000; Full time

Job summary

Palo Alto Networks is seeking a Senior Data Platform Engineer to design and build scalable data pipelines. You will collaborate with Product, SRE, ML Engineering, and Security to deliver high-throughput streaming services and resilient batch/data workflows.

You’ll profile and tune distributed pipelines, optimize cloud costs, and advocate best practices in engineering across the team. A strong background in Golang, Python, and cloud-native data stacks is required.

Qualifications

• Bachelor’s degree in Computer Science or a related field with 10+ years of relevant industry experience, or a Master’s degree with 8+ years of experience.
• Deep domain expertise in designing, building, and operating highly scalable, fault‑tolerant distributed systems, with exceptional programming skills in Golang and Python.
• Hands‑on experience building backend data platform infrastructure using distributed execution engine such as Apache Beam or Apache Spark.
• Hands‑on experience architecting, building, or deploying complex agentic workflows, tool‑use pipelines, or multi‑agent orchestrations within data infrastructure.
• Proven track record of profiling, benchmarking, and tuning high‑throughput distributed pipelines to improve latency and resource efficiency.
• Broad experience with cloud‑native data stacks (GCP/Dataflow preferred), messaging (Pub/Sub, Kafka), storage engines (BigQuery, Iceberg), and infrastructure tooling (Kubernetes, Terraform).
• Demonstrated ability to drive consensus across multi‑team boundaries, solve complex technical problems, and mentor senior engineering talent.
• Active contributions or committer status in open‑source big data projects (Apache Beam, Spark, Airflow, Iceberg, etc.).

Responsibilities

• Partner with Product Management, SRE, ML Engineering, Security Researchers, and Quality Engineering to design and deliver high‑throughput data platform capabilities and streaming services.
• Architect and build resilient, multi‑source batch and streaming engines to power analytical workflows, business insights, and internal feature improvements.
• Profile, benchmark, and tune distributed execution DAGs, state storage, and compute resources to maximize throughput, lower processing latency, and optimize cloud expenditure at scale.
• Evangelize and implement modern software engineering best practices, including spec‑driven development, comprehensive testing frameworks, and agentic coding workflows.
• Act as a force multiplier by mentoring senior and junior engineers, conducting rigorous architectural reviews, and cultivating a culture of technical excellence across the team.

Skills

Golang

Python

Distributed systems

Backend data platforms

Performance optimization

Cloud data stacks

Terraform

Kubernetes

Data engineering

Open-source contributions

Education

Bachelor’s degree in Computer Science or related field

Master’s degree

Tools

Apache Beam

Apache Spark

Pub/Sub

Kafka

Kubernetes

Terraform

BigQuery

Iceberg

Dataflow

Job description

Palo Alto Networks is seeking a Senior Data Platform Engineer to design and build scalable data pipelines. You will collaborate with Product, SRE, ML Engineering, and Security to deliver high-throughput streaming services and resilient batch/data workflows.

You’ll profile and tune distributed pipelines, optimize cloud costs, and advocate best practices in engineering across the team. A strong background in Golang, Python, and cloud-native data stacks is required.