Senior Data Engineer – Real-Time Data Platform (AWS, Spark)

Oliver Bernard; Greater London; Hybrid; GBP 129,150 - 166,050; Full time

Job summary

Oliver Bernard is hiring a Senior Data Engineer to lead design and delivery of a modern, cloud-native data platform in a large financial services environment. The role blends architecture, engineering leadership, and hands-on development across real-time and batch data platforms built on Python, AWS and Spark.

You will guide data governance, security, and audit-compliant solutions while mentoring multiple data engineering squads within a hybrid London-based setting (Canary Wharf 3 days onsite).

Qualifications

• Strong Spark experience building streaming platforms.
• Hands-on experience with Apache Kafka for real-time data pipelines.
• Extensive AWS exposure (S3, Lambda, ECS/EKS) and data governance awareness.

Responsibilities

• Own the design and evolution of the enterprise data platform.
• Define best practices, engineering standards, and data governance patterns.
• Provide technical leadership to multiple data engineering squads.
• Drive adoption of event-driven and streaming architectures.
• Design and build high-throughput, low-latency data pipelines.
• Implement robust real-time streaming solutions using Kafka.
• Develop and maintain scalable Python-based data services.
• Architect secure and resilient AWS cloud data infrastructure.
• Ensure solutions meet regulatory, security and audit requirements.

Skills

Spark

Apache Kafka

AWS

Data modelling

Event-driven architecture

Microservices

Regulated financial services

Tools

Terraform

Docker

Kubernetes

CI/CD tooling

Job description

Oliver Bernard is hiring a Senior Data Engineer to lead design and delivery of a modern, cloud-native data platform in a large financial services environment. The role blends architecture, engineering leadership, and hands-on development across real-time and batch data platforms built on Python, AWS and Spark.

You will guide data governance, security, and audit-compliant solutions while mentoring multiple data engineering squads within a hybrid London-based setting (Canary Wharf 3 days onsite).