Senior Data Engineer - Scalable AI Data Platform

The BDPN; Greater London; Hybrid; GBP 110,000 - 160,000; Full time

Job summary

The BDPN is seeking a Principal Data Engineer to drive the technical direction of our data platform across ingestion, transformation, storage and governance. You will design and implement scalable pipelines that process large data volumes in real-time and batch, enabling stakeholders to build advanced analytics and AI features.

You will lead architecture and deployment of Big Data systems, evaluate new tools, and mentor engineering teams while partnering with analytics, product and platform

Qualifications

• Experience building large-scale data platforms in production.
• Strong knowledge of distributed computing and data architecture.
• Hands-on with ETL/ELT pipelines and data governance.

Responsibilities

• Lead data architecture and deployment of scalable Big Data systems.
• Develop end-to-end ETL/ELT pipelines and data ingestion frameworks.
• Design data lakes, warehouses, and real-time streaming architectures.
• Collaborate with analytics, product, and platform teams on data models.
• Implement data quality, governance, and observability at scale.

Skills

Data engineering

Python

SQL

Cloud platforms

Tools

Apache Spark

Flink

Hadoop

Kafka

Kinesis

Airflow

Dagster

Snowflake

BigQuery

Redshift

Databricks

Delta Lake

Apache Iceberg

Job description

The BDPN is seeking a Principal Data Engineer to drive the technical direction of our data platform across ingestion, transformation, storage and governance. You will design and implement scalable pipelines that process large data volumes in real-time and batch, enabling stakeholders to build advanced analytics and AI features.

You will lead architecture and deployment of Big Data systems, evaluate new tools, and mentor engineering teams while partnering with analytics, product and platform