Staff Data Engineer — AI-Driven Data Platform Lead

United States Digital Space LLC; Greater London; Hybrid; GBP 130,000 - 170,000; Full time

Benefits offered by this job

Equity

Learning budget

Wellness reimbursement

Parental leave

Team offsites

Job summary

Do Your Best Work in London-based hybrid settings seeks a Staff Software Engineer for the Data Engineering team to steer the Caspian Data Platform. You’ll own ingestion, transformation, governance, and data quality across the lakehouse, shaping architecture while staying deeply hands-on.

You’ll set standards for AI-enabled pipelines, guide senior engineers, and define production patterns while delivering high-impact data capabilities across the business units.

Qualifications

• 10+ years of data engineering experience with hands-on architecture and operations at scale.
• Mastery of Databricks stack and platform-level decision making.
• Ownership of ingestion, transformation, governance, and data quality in a production platform.
• Expertise in SQL and Python applied to data modeling and tooling.
• Strong AWS experience operating data systems at scale.
• Ability to set standards, patterns, and raise overall code quality.

Responsibilities

• Lead design of core platform pillars: ingestion, transformation, governance, and data quality, building key components.
• Define engineering patterns and reference architectures for Databricks pipelines; ship production flows hands-on.
• Set benchmarks for data quality and governance, including contracts, lineage, and validation frameworks.
• Own complex initiatives spanning multiple quarters and teams, from problem statement to delivered capability.
• Lead technical direction for applying AI within data engineering; prototype agentic systems and self-serve analytics.
• Raise code quality and engineering judgment; mentor senior engineers for greater ownership.

Skills

Databricks

Delta Live Tables

Unity Catalog

Delta Lake

Spark

SQL

Python

AWS

Job description

Do Your Best Work in London-based hybrid settings seeks a Staff Software Engineer for the Data Engineering team to steer the Caspian Data Platform. You’ll own ingestion, transformation, governance, and data quality across the lakehouse, shaping architecture while staying deeply hands-on.

You’ll set standards for AI-enabled pipelines, guide senior engineers, and define production patterns while delivering high-impact data capabilities across the business units.