Lead Data Engineer

MRJ Recruitment; Manchester; Hybrid; GBP 95.000 - 105.000; Vollzeit

Zusammenfassung

MRJ Recruitment is seeking a Lead Data Engineer in Manchester or Warrington with a hybrid working model. You will own the technical direction and drive the transformation of a growing data platform, including a large-scale migration from on-prem SQL Server to an AWS-based lakehouse.

You’ll lead architectural decisions, set engineering standards and mentor the team, while staying hands-on with Python, PySpark and SQL, and shaping streaming, data quality and observability practices.

Qualifikationen

• Significant experience in Data Engineering with leadership of large-scale data platform or data warehouse migrations.
• Advanced Python, PySpark and SQL skills are essential.
• Experience building real-time streaming architectures using Kafka or MSK.
• Strong understanding of data warehousing, data modelling and lakehouse concepts.
• Experience with open table formats like Iceberg or Delta Lake.
• Infrastructure-as-code and CI/CD experience (Terraform preferred).

Aufgaben

• Lead the technical delivery of a major data warehouse migration from on-prem SQL Server to AWS-based lakehouse.
• Define architecture, engineering standards and design patterns for the data platform.
• Design and develop scalable batch and real-time data processing solutions.
• Build and improve data pipelines for reliability and efficiency.
• Develop real-time streaming capabilities to reduce latency in processing and reporting.
• Establish best practices around data modelling, quality, validation and observability.
• Own infrastructure-as-code and CI/CD practices across the platform.

Kenntnisse

AWS data services

Python

PySpark

SQL

Apache Kafka / MSK

Data warehousing

Lakehouse architecture

Terraform

CI/CD

ETL pipelines

Tools

Terraform

Apache Kafka

Amazon MSK

Glue

Lambda

S3

Redshift

Jobbeschreibung

Lead Data Engineer | £95,000 - £105,000 | Manchester or Warrington (Hybrid)

We're looking for a Lead Data Engineer to take ownership of the technical direction and continued development of a growing data platform.

A major part of the role will be leading the technical delivery of a large-scale data warehouse migration, moving an existing on-premise SQL Server environment towards a modern AWS-based lakehouse architecture.

You'll play a key role in shaping the platform's architecture, establishing engineering standards and driving improvements across data processing, streaming and overall platform performance.

This is a genuinely hands‑on technical leadership position. You'll have the autonomy to make architectural decisions, challenge existing approaches and guide the wider engineering team, while remaining close to the code and day‑to‑day technical delivery.

What You’ll Be Doing

• Leading the technical delivery of a major data warehouse migration from on-premise SQL Server to an AWS-based lakehouse.
• Defining the architecture, engineering standards and design patterns for the wider data platform.
• Designing and developing scalable batch and real‑time data processing solutions.
• Building and improving data pipelines, ensuring they are reliable, efficient and maintainable.
• Developing real‑time streaming capabilities to significantly reduce data processing and reporting latency.
• Establishing best practices around data modelling, data quality, validation and platform observability.
• Taking ownership of infrastructure‑as‑code and CI/CD standards across the platform.

What We’re Looking For

• Significant experience in Data Engineering, with a proven track record of leading large‑scale data platform or data warehouse migrations.
• Deep, hands‑on experience with AWS data services, particularly Glue, Lambda, S3, Redshift and EMR.
• Advanced Python, PySpark and SQL skills.
• Proven experience building and operating real‑time streaming architectures using Apache Kafka or Amazon MSK.
• Strong understanding of data warehousing, data modelling and modern lakehouse architecture.
• Experience with open table formats such as Apache Iceberg or Delta Lake.
• Experience with infrastructure‑as‑code, ideally Terraform, alongside CI/CD and pipeline orchestration.