Staff Software Engineer, Datalake Platform

EngineersOfAI; Dublin; On-site; EUR 70,000 - 100,000; Full time

Job summary

EngineersOfAI is seeking a skilled architect to lead the unified Iceberg platform's design and implementation. This role involves defining API contracts, managing the metastore migration strategy, and shaping object storage policies.

As part of a high-impact team, you will influence data management at petabyte scale, ensuring compliance and optimizing costs.

Qualifications

• Strong experience in architecting data management systems.
• Ability to lead technical teams in high-impact projects.
• Familiarity with data governance and regulatory compliance.

Responsibilities

• Architect the unified Iceberg platform for metadata management.
• Drive the migration strategy for metastore workloads.
• Define the object storage abstraction and policies.
• Partner with security for compliance architecture.
• Identify inefficiencies at petabyte scale and automate solutions.
• Mentor senior engineers through design reviews.

Skills

Architectural design

Data governance

Technical leadership

Compliance architecture

Storage systems

Job description

The Team

The Datalake team builds and maintains Stripe's foundational data access and governance infrastructure. It serves developers, data engineers, analysts, ML and AI teams, security teams, and business users across the company.

What Makes This Role Compelling

• Foundational infrastructure with broad reach: systems sit in the critical path of nearly every data workload, affecting petabytes of data and hundreds of production pipelines.
• Active, high‑stakes architectural transformation: leading a multi‑year migration to modern, OSS‑aligned solutions.
• Storage platform ownership with room to define the approach: shaping the object storage abstraction layer and its policies.
• Large‑scale impact: influencing how hundreds of engineering teams interact with petabytes of data.

Responsibilities

• Architect the unified Iceberg platform: lead the technical design of a metastore service that becomes the single source of truth for Iceberg table management across all compute engines, defining API contracts, authorization models, and integration patterns.
• Own the metastore migration strategy: drive sequencing, backward compatibility, rollback approaches, and cross‑team coordination for migrating all remaining Hive Metastore‑backed workloads.
• Shape the object storage abstraction: define the storage abstraction layer, including bucket provisioning, access control policy design, and developer‑facing client libraries.
• Lead compliance architecture: partner with security and compliance teams to translate regulatory requirements into technical controls such as audit logging, access review infrastructure, data segregation, and lifecycle enforcement.
• Drive cost and efficiency at petabyte scale: identify inefficiencies in storage layout, snapshot retention, and data lifecycle, and design automated, self‑service tooling that scales without ongoing manual intervention.
• Set the technical bar: own critical design reviews, establish standards for reliability, security, and developer experience, and mentor senior engineers through high‑stakes architectural decisions.