Senior Staff Engineer, Big Data Infrastructure

Wwshemi; San Francisco (CA); On-site; USD 250,000 - 400,000; Full time

Job summary

Stripe San Francisco is seeking a Senior Staff engineer to lead the Big Data Infrastructure team within Core Compute. You’ll guide a 40+ engineering organization and partner with Data Foundations, Payins Eng, and AI/ML Foundations to modernize data pipelines and scale platforms powering thousands of engineers.

The role focuses on multi-region/multi-cloud data strategy, migrations (YARN to Kubernetes), vectorization, Airflow, and data catalogs, with a strong emphasis on reliability, security, and

Qualifications

• 12+ years of professional software development experience
• Strong background in distributed infrastructure/platform services
• Experience leading cross-team, company-wide projects with minimal downtime

Responsibilities

• Serve as technical leader for Big Data Infrastructure, overseeing 40+ engineers
• Lead modernization bets (YARN to Kubernetes, native vectorization)
• Drive multi-region and multi-cloud data strategy for high availability and DR
• Oversee data pipelines from authoring to operation by AI agents
• Reduce KTLO by shifting fixes into platform rather than runbooks
• Mentor engineers across career stages, up to Staff+

Skills

Distributed infra

Platform services

Cross-team leadership

Data pipelines

Reliability engineering

Cost optimization

Education

BS/MS in Computer Science or related field

Tools

AWS

Azure

Google Cloud

Kubernetes

Airflow

Data catalogs

Job description

Stripe San Francsico Posted 2026-11-05

This listing's locations San Francsico

Department: Core Compute

About the team

In this role, you’d be joining the Big Data Infrastructure team. Our team builds and maintains the platforms that power data work across Stripe, managing three core layers:

• Data Orchestration (Airflow)
• Batch Compute (Spark, Celeborn)

We support over 2,000 engineers and 600+ internal teams, processing workloads that organically grow more than 30% year over year. Our infrastructure handles massive scale, managing over 800 PB of data, running 325,000 applications a day, and accounting for hundreds of millions of dollars in annual spend, directly powering critical pipelines like inbound payments, network reconciliation, and capital underwriting.

What you’ll do

As the technical leader for the Big Data Infrastructure organization, you will guide a 40+ person engineering team to scale our foundational data systems. You will partner closely with major internal customers like Data Foundations, Payins Engineering, and AI/ML Foundations to modernize our tech stack and execute Stripe's transition to the "Agentic Era" - building "invisible infrastructure" where AI agents operate data pipelines end-to-end.

Responsibilities

• Serve as the technical leader for the Big Data Infrastructure organization, overseeing distributed systems that manage 20% of Stripe's total AWS spend
• Execute Stripe's transition to the "Agentic Era" by building invisible infrastructure that empowers AI agents to author, validate, ship, and operate data pipelines
• Drive the architecture and delivery of critical modernization bets, including YARN to Kubernetes migrations, native vectorization (Gluten/Velox), and Airflow and Data Catalog modernization
• Lead the implementation of a multi-region and multi-cloud data strategy for high availability, disaster recovery, and Azure Blob Storage compliance
• Reduce operational burden and maintenance overhead (KTLO) from over 40% to under 10% by shifting fixes into the platform instead of relying on runbooks
• Foster a culture of operational excellence and mentor engineers at various stages in their careers, up to Staff+

Who you are

We’re looking for someone who meets the minimum requirements to be considered for the role. If you meet these requirements, you are encouraged to apply. The preferred qualifications are a bonus, not a requirement.

• BS or MS in Computer Science or a related field
• This is a Senior Staff role, so ideally 12+ years of professional experience in a software development role
• A strong engineering background in distributed infrastructure and/or platform services, including one or more of the following: storage, databases, or compute
• Experience leading projects on a cross-team and company-wide scale, including scaling and migrating systems with little to no downtime
• Experience developing, maintaining, and debugging distributed systems while optimizing their reliability and security
• Experience driving a culture of operational excellence, with a deep understanding of metrics, alerts, and dashboards

Preferred qualifications

• Experience driving AI-led architectural transformations and integrating agentic workflows into large-scale data systems
• Experience in a development team at AWS, Azure, OCI, or Google
• Experience with big data infrastructure at scale, such as Spark, Iceberg, S3, Airflow, YARN, and Celeborn
• Experience with database internals and/or with Kubernetes