Senior Staff Engineer, Big Data Infrastructure Staff

D8 LooP; San Francisco (CA); On-site; USD 180,000 - 240,000; Full time

Job summary

Stripe is seeking a leadership-minded data platform engineer to drive modernization across big data stacks. You will oversee distributed systems, optimize reliability and security, and lead initiatives from YARN to Kubernetes migrations and data catalog modernization.

You will work with Spark, Iceberg, S3 and related technologies to manage a scalable data platform while driving operational excellence through robust metrics, alerts, and dashboards.

Qualifications

• Must have experience driving modernization bets including YARN to Kubernetes migrations and data catalog modernization.
• Must have Spark and big data infra at scale (Spark, Iceberg, S3, Airflow, YARN, Celeborn).
• Experience leading distributed systems with reliability and security in production.
• Proven track record of driving operational excellence with metrics, alerts and dashboards.

Skills

Airflow orchestration

Spark

AWS

Distributed systems

Operational excellence

YARN

Kubernetes

Data governance

SQL

Tools

Airflow

Iceberg

S3

YARN

Gluten

Velox

Celeborn

Job description

San Francsico Staff 12+ yrs Data Platform posted 1 d ago

Stripe is a financial infrastructure platform for businesses. Millions of companies - from the world’s largest enterprises to the most ambitious startups - use Stripe to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead.

Skills, with evidence

• Airflow / orchestration Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations, native vectorization (Gluten/Velox), and Airflow and Data Catalog modernization must have
• Spark Experience with big data infrastructure at scale, such as Spark, Iceberg, S3, Airflow, YARN, and Celeborn must have
• AWS Serve as the technical leader for the Big Data Infrastructure organization, overseeing distributed systems that manage 20% of Stripe's total AWS spend must have · not practised here
• Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations, native vectorization (Gluten/Velox), and Airflow and Data Catalog modernization must have · not practised here
• Experience developing, maintaining, and debugging distributed systems while optimizing their reliability and security must have
• Experience driving a culture of operational excellence, with a deep understanding of metrics, alerts, and dashboards
• Data Lake (Iceberg, Hive, S3, Azure Blob Storage) not practised here
• Warehousing Data Lake (Iceberg, Hive, S3, Azure Blob Storage)
• Governance & security Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations, native vectorization (Gluten/Velox), and Airflow and Data Catalog modernization not practised here
• SQL Data Lake (Iceberg, Hive, S3, Azure Blob Storage)
• Experience driving a culture of operational excellence, with a deep understanding of metrics, alerts, and dashboards not practised here