Senior Data Engineer

Understanding Recruitment Ltd; Greater London; On-site; GBP 85,000 - 120,000; Full time

Job summary

Understanding Recruitment Ltd is hiring a Senior Data Engineer to take ownership of a modern, AWS-native data platform in London. You'll shape data ingestion, transformation and scalable pipelines powering production ML and AI initiatives.

This hands-on role sits in a lean engineering team, collaborating with data scientists, ML engineers and architects to ensure reliable, high-quality data across batch and streaming workloads, and to evolve the data lake architecture.

Qualifications

• Hands-on experience building production data engineering systems.
• Deep familiarity with AWS and cloud-native data architectures.
• Strong Python and SQL skills with solid software engineering fundamentals.
• Experience designing and maintaining scalable ETL/ELT pipelines and data lakes.
• Understanding of distributed data processing and streaming architectures.
• Ability to take end-to-end ownership and solve technical problems autonomously.
• Experience with Apache Iceberg, Kafka, Parquet or Athena is highly desirable; SageMaker exposure is a bonus.

Responsibilities

• Design, build and maintain scalable, production-grade data pipelines across batch and streaming workloads.
• Evolve the AWS data lake architecture to improve data quality, reliability and performance.
• Build and optimise data ingestion, transformation and processing workflows with modern tools.
• Collaborate with data scientists, ML engineers and architects to ensure high-quality data for production ML systems.
• Influence architectural decisions and establish engineering best practices for the data platform.

Skills

Python

SQL

Data engineering

End-to-end ownership

Tools

Apache Iceberg

Kafka

Parquet

Athena

SageMaker

Job description

Senior Data Engineer

We're hiring aSenior Data Engineerto take technical ownership of a modern, AWS-native data platform, spanning everything from raw data ingestion and transformation to building scalable, reliable data pipelines that power production machine learning and AI systems.

This is a hands-on, high-autonomy role within a lean engineering team, where you'll work directly with senior technical leadership and have a genuine say in how the data architecture and platform evolve.

The Engineering Challenge

You’ll own and evolve an AWS-native data environment, improving the reliability, scalability and performance of existing data infrastructure while building the foundations for future ML and AI initiatives.

The current stack includes:

• Data Engineering:Apache Iceberg, Kafka, Parquet and Amazon Athena
• Cloud Infrastructure:AWS-native data lake architecture, data ingestion, processing and storage
• Architecture:AWS-based data lake (~5TB), with opportunities to modernise pipelines, optimise performance and improve platform maturity
• ML & AI Integration:Supporting production ML workflows through SageMaker, with future opportunities around GenAI and LLM-powered diagnostic capabilities

What You’ll Own

• Designing, building and maintaining scalable, production-grade data pipelines across batch and streaming workloads
• Evolving the AWS data lake architecture, improving data quality, reliability, performance and accessibility
• Building and optimising data ingestion, transformation and processing workflows using modern data engineering technologies
• Working directly with data scientists, ML engineers and architects to ensure reliable, high-quality data is available for production ML systems
• Influencing architectural decisions, establishing engineering best practices and shaping the next generation of data infrastructure

What We’re Looking For

• Strong hands-on experience building and operatingproduction data engineering systems
• Deep familiarity withAWS and cloud-native data architectures
• Strong Python and SQL skills, with solid software engineering fundamentals
• Experience designing and maintaining scalable ETL/ELT pipelines, data lakes or modern lakehouse architectures
• Understanding of distributed data processing, streaming architectures and data pipeline optimisation
• Someone who enjoys end-to-end ownership, technical problem-solving and working without layers of bureaucracy
• Experience withApache Iceberg, Kafka, Parquet or Athenais highly desirable, while exposure to SageMaker or ML infrastructure is a bonus -production data engineering expertise is the priority.