Senior Big Data Infrastructure Architect

D8 LooP; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Stripe is seeking a leadership-minded data platform engineer to drive modernization across big data stacks. You will oversee distributed systems, optimize reliability and security, and lead initiatives from YARN to Kubernetes migrations and data catalog modernization.

You will work with Spark, Iceberg, S3 and related technologies to manage a scalable data platform while driving operational excellence through robust metrics, alerts, and dashboards.

Qualifications

• Must have experience driving modernization bets including YARN to Kubernetes migrations and data catalog modernization.
• Must have Spark and big data infra at scale (Spark, Iceberg, S3, Airflow, YARN, Celeborn).
• Experience leading distributed systems with reliability and security in production.
• Proven track record of driving operational excellence with metrics, alerts and dashboards.

Skills

Airflow orchestration

Spark

AWS

Distributed systems

Operational excellence

YARN

Kubernetes

Data governance

SQL

Tools

Airflow

Iceberg

S3

YARN

Gluten

Velox

Celeborn

Job description

Stripe is seeking a leadership-minded data platform engineer to drive modernization across big data stacks. You will oversee distributed systems, optimize reliability and security, and lead initiatives from YARN to Kubernetes migrations and data catalog modernization.

You will work with Spark, Iceberg, S3 and related technologies to manage a scalable data platform while driving operational excellence through robust metrics, alerts, and dashboards.