Lead Data Engineer - AWS Lakehouse & Real-Time Pipelines

MRJ Recruitment; Manchester; Hybrid; GBP 95,000 - 105,000; Full time

Job summary

MRJ Recruitment is seeking a Lead Data Engineer in Manchester or Warrington with a hybrid working model. You will own the technical direction and drive the transformation of a growing data platform, including a large-scale migration from on-prem SQL Server to an AWS-based lakehouse.

You’ll lead architectural decisions, set engineering standards and mentor the team, while staying hands-on with Python, PySpark and SQL, and shaping streaming, data quality and observability practices.

Qualifications

• Significant experience in Data Engineering with leadership of large-scale data platform or data warehouse migrations.
• Advanced Python, PySpark and SQL skills are essential.
• Experience building real-time streaming architectures using Kafka or MSK.
• Strong understanding of data warehousing, data modelling and lakehouse concepts.
• Experience with open table formats like Iceberg or Delta Lake.
• Infrastructure-as-code and CI/CD experience (Terraform preferred).

Responsibilities

• Lead the technical delivery of a major data warehouse migration from on-prem SQL Server to AWS-based lakehouse.
• Define architecture, engineering standards and design patterns for the data platform.
• Design and develop scalable batch and real-time data processing solutions.
• Build and improve data pipelines for reliability and efficiency.
• Develop real-time streaming capabilities to reduce latency in processing and reporting.
• Establish best practices around data modelling, quality, validation and observability.
• Own infrastructure-as-code and CI/CD practices across the platform.

Skills

AWS data services

Python

PySpark

SQL

Apache Kafka / MSK

Data warehousing

Lakehouse architecture

Terraform

CI/CD

ETL pipelines

Tools

Terraform

Apache Kafka

Amazon MSK

Glue

Lambda

S3

Redshift

Job description

MRJ Recruitment is seeking a Lead Data Engineer in Manchester or Warrington with a hybrid working model. You will own the technical direction and drive the transformation of a growing data platform, including a large-scale migration from on-prem SQL Server to an AWS-based lakehouse.

You’ll lead architectural decisions, set engineering standards and mentor the team, while staying hands-on with Python, PySpark and SQL, and shaping streaming, data quality and observability practices.