Senior Data Engineers

Oliver Bernard; Greater London; Hybrid; GBP 129,150 - 166,050; Full time

Job summary

Oliver Bernard is hiring a Senior Data Engineer to lead design and delivery of a modern, cloud-native data platform in a large financial services environment. The role blends architecture, engineering leadership, and hands-on development across real-time and batch data platforms built on Python, AWS and Spark.

You will guide data governance, security, and audit-compliant solutions while mentoring multiple data engineering squads within a hybrid London-based setting (Canary Wharf 3 days onsite).

Qualifications

• Strong Spark experience building streaming platforms.
• Hands-on experience with Apache Kafka for real-time data pipelines.
• Extensive AWS exposure (S3, Lambda, ECS/EKS) and data governance awareness.

Responsibilities

• Own the design and evolution of the enterprise data platform.
• Define best practices, engineering standards, and data governance patterns.
• Provide technical leadership to multiple data engineering squads.
• Drive adoption of event-driven and streaming architectures.
• Design and build high-throughput, low-latency data pipelines.
• Implement robust real-time streaming solutions using Kafka.
• Develop and maintain scalable Python-based data services.
• Architect secure and resilient AWS cloud data infrastructure.
• Ensure solutions meet regulatory, security and audit requirements.

Skills

Spark

Apache Kafka

AWS

Data modelling

Event-driven architecture

Microservices

Regulated financial services

Tools

Terraform

Docker

Kubernetes

CI/CD tooling

Job description

Location: 3 days onsite in Canary Wharf, London

Overview

We are seeking Senior Data Engineers to join a large financial services organisation undergoing a major data platform transformation. This role sits at the intersection of architecture, engineering leadership, and hands-on development, shaping a modern, cloud-native data ecosystem.

You will lead the design and delivery of highly scalable, real-time and batch data platforms built on Python, AWS and Spark, working closely with architecture, trading, risk, and analytics teams.

Key Responsibilities

• Own the design and evolution of the enterprise data platform
• Define best practices, engineering standards, and data governance patterns
• Provide technical leadership to multiple data engineering squads
• Drive adoption of event-driven and streaming architectures
• Design and build high-throughput, low-latency data pipelines
• Implement robust real-time streaming solutions using Kafka
• Develop and maintain scalable Python-based data services
• Architect secure and resilient AWS cloud data infrastructure
• Ensure solutions meet regulatory, security and audit requirements

Stakeholder Engagement

• Partner with senior stakeholders across Risk, Trading, Compliance and Data Science
• Translate complex business requirements into scalable technical solutions
• Mentor senior and mid-level engineers across the programme

Required Skills & Experience

Core Technical

• Strong Spark
• Deep experience building streaming platforms using Apache Kafka
• Strong hands-on experience with AWS, including:
• S3, Lambda, ECS/EKS
• IAM, networking and security best practices
• Experience building real-time and batch ETL/ELT pipelines
• Strong experience with data modelling and distributed systems
• Strong understanding of event-driven and microservices architectures
• Experience working in regulated financial services environments
• Experience with data mesh / data lakehouse architectures
• Knowledge of CI/CD and DevOps tooling (Terraform, Docker, Kubernetes)
• Experience with data governance and lineage tooling
• Exposure to market, risk, or trading data environments

Working Pattern

• Hybrid: 3 days onsite in Canary Wharf, 2 days remote
• £800 per day inside IR35
• 12-month contract with strong likelihood of extension