Staff Engineer, Datalake Platform

United States Digital Space LLC; Dublin; On-site; EUR 90,000 - 130,000; Full time

Job summary

United States Digital Space LLC, located in Dublin, is seeking a Senior Engineer to architect a unified Iceberg platform for data management. You will lead complex migration strategies and define policies for object storage solutions used across the company's data systems.

Candidates should have extensive experience in software engineering, with a strong focus on data infrastructure at petabyte scale. The role offers significant influence over technical strategy and the roadmap for a large-scale data ecosystem.

Qualifications

• 10+ years of professional software engineering experience.
• Experience designing and operating large-scale distributed storage systems.
• Deep experience with object storage systems and IAM.

Responsibilities

• Architect the unified Iceberg platform for data management.
• Drive the metastore migration strategy across teams.
• Define storage abstraction layer and access control policy.
• Lead compliance architecture with security teams.
• Optimize storage efficiencies and automate tooling.
• Mentor and set technical standards for the engineering team.

Skills

Object storage expertise (S3, Azure Blob)

Experience with distributed data systems

Project management skills

Authorization and access control design

Problem-solving at petabyte scale

Job description

Who We Are

the company is a financial infrastructure platform for businesses. Millions of companies — from the world's largest enterprises to the most ambitious startups — use the company to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead. That means you have an unprecedented opportunity to put the global economy within everyone's reach while doing the most important work of your career.

About the company

the company is a financial infrastructure platform for businesses. Millions of companies — from the world's largest enterprises to the most ambitious startups — use the company to accept payments, grow their revenue, and accelerate new business opportunities. Our mission is to increase the GDP of the internet, and we have a staggering amount of work ahead. That means you have an unprecedented opportunity to put the global economy within everyone's reach while doing the most important work of your career.

About the Team

The Datalake team builds and maintains the company's foundational data access and governance infrastructure — the paved path for safe, fast, and compliant access to the company's critical big data assets. We serve developers, data engineers, analysts, ML and AI teams, security teams, and business users across the company. The team is in the middle of a significant architectural transition as the company grows. We are making the company's data lake a first-class citizen of the modern data ecosystem to support our growing scale and diverse workloads.

What Makes This Role Compelling

• Foundational infrastructure with broad reach: The Datalake team's systems sit in the critical path of nearly every data workload at the company. Decisions affect petabytes of data, hundreds of production pipelines, and every engineering team that builds on the company's data lake.
• Active, high-stakes architectural transformation: The team is executing a multi-year migration to modern, OSS-aligned solutions — a technically deep project with real architectural choices at each step, including API design, compute engine integration, authorization model, and per-table credential vending.
• Active, high-stakes, OSS-aligned architectural transformation: You will lead a multi-year migration to modern, open-source solutions like the Apache Iceberg REST Catalog. This is a technically deep project involving critical architectural choices at each step, from API design and compute engine integration to authorization models, where your opinions and technical influence will directly shape how the platform engages with the broader data infrastructure ecosystem.
• Storage platform ownership with room to define the approach: The team owns the object storage abstraction layer — access control, IAM policy design, lifecycle management, and compliance architecture — but the how is still being written. You'll shape how hundreds of engineering teams interact with petabytes of data, and the decisions you make will stick.
• At the company you’ll have the scale of the large company and the agency to influence technical strategy and the roadmap

Responsibilities

• Architect the unified Iceberg platform: Lead the technical design of a metastore service as it becomes the single source of truth for Iceberg table management across all compute engines — Spark, Trino, Flink, and PyIceberg. Define the API contracts, authorization model, per-table credential vending, and integration patterns that every data pipeline at the company will depend on.
• Own the metastore migration strategy: Drive the sequencing, backward compatibility story, rollback approach, and cross-team coordination for migrating all remaining Hive Metastore-backed workloads to the new platform. This means coordinating with dozens of consuming teams while keeping production data infrastructure operational at all times.
• Shape the object storage abstraction: Define the storage abstraction layer — including bucket provisioning, access control policy design, and the developer-facing client libraries that make object storage ergonomic and secure by default. The goal is an abstraction layer that consuming teams can adopt without needing to become storage infrastructure experts themselves.
• Lead compliance architecture: Partner with security and compliance teams to translate regulatory requirements into durable preventative technical controls — audit logging, access review infrastructure, data segregation, and lifecycle enforcement — built into the platform rather than bolted on.
• Drive cost and efficiency at petabyte scale: Identify systemic inefficiencies in storage layout, snapshot retention, and data lifecycle, and design automated, self-service tooling that scales without ongoing manual intervention from the team.
• Set the technical bar: Own critical design reviews, establish standards for reliability, security, and developer experience, and mentor senior engineers through high-stakes architectural decisions. Provide the technical judgment that keeps the platform moving fast without accumulating structural debt.

Who You Are

Minimum requirements

• 10+ years of professional software engineering experience.
• Demonstrated track record of designing, building, and operating large-scale distributed storage or data infrastructure systems.
• Deep experience with object storage (S3, Azure Blob, or equivalent) — including IAM, access control policy design, lifecycle management, and operational practices at petabyte scale.
• Proven ability to lead complex, multi-quarter infrastructure projects end-to-end, including cross-team dependency management and coordinating migrations across many consuming teams.
• Strong background in authorization and access control design for distributed data systems.

Preferred requirements

• Deep expertise in Apache Iceberg — table format internals, the REST Catalog specification, snapshot lifecycle management, compaction, and compute engine integration (Spark, Trino, Flink, PyIceberg).
• Background in compliance-sensitive infrastructure — SOX, ICFR, or equivalent regulatory frameworks — with an understanding of how audit and access review requirements translate into preventative technical controls.
• Experience safely executing large-scale data migrations with a strong instinct for sequencing, blast radius reduction, rollback, and data integrity validation.
• A strong developer experience sensibility: the ability to build abstractions that are ergonomic, well-documented, and actively reduce toil for the engineering teams that depend on your platform.