Lead Data Engineer - Multi-Cloud Databricks & Data Platform

Serco

Metropolitan Borough of Solihull, Greater London, Hook

Hybrid

GBP 75,000 - 85,000

Full time

7 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Company car
5% on target bonus scheme
Private medical insurance
Pension – 6%
Chance to contribute to innovation in/
A company passionate about diversity
Serco discounts
Employee wellbeing benefits
Career development training
Safe and supportive culture

Job summary

Serco is seeking a Lead Data Engineer to act as the technical authority for data engineering across a multi-cloud environment (Azure, AWS and Databricks). The role blends hands-on design and delivery (about 80%) with leadership of engineers, stakeholder influence and shaping data engineering practices.

You will champion Databricks as the core platform, govern data models, and drive cost-efficient, scalable solutions while collaborating across a matrix organisation to deliver enterprise-grade

Qualifications

  • Expert-level experience in enterprise-scale data engineering, including defining standards, patterns, and overseeing complex delivery.
  • Advanced proficiency in PySpark and SQL, with deep expertise in performance optimisation and scalable data processing.
  • Strong experience in data modelling and large-scale data integration design, with the ability to build robust, maintainable, and testable pipelines.
  • Proven ability to embed testing frameworks, observability, and data quality controls across engineering workflows.
  • Deep hands-on experience with Databricks (essential), including lakehouse architecture, Delta Lake, Unity Catalog, and streaming technologies.
  • Broad cloud expertise across Azure and AWS, with strong multi-cloud experience.
  • Strong understanding of modern engineering practices, including CI/CD for data platforms, automated deployment pipelines, and environment promotion strategies.
  • Experience designing monitoring, observability, and cost optimisation frameworks, alongside governance, lineage, cataloguing, and compliance at scale.
  • Excellent stakeholder management skills, with the ability to influence senior leaders and work effectively across matrixed organisations.
  • Proven leadership capability, including mentoring, coaching, and developing engineering teams.
  • Strong communication skills, with the ability to translate complex technical concepts into clear business value.
  • 8+ years’ experience in data engineering within enterprise environments.
  • Relevant certifications (desirable) such as Azure Data Engineer Associate, AWS Data Analytics Specialty, or Databricks Data Engineer (Associate/Professional).
  • SC clearance (or willingness to obtain it)

Responsibilities

  • Define and enforce data engineering standards, patterns, and best practices across the organisation.
  • Architect and deliver scalable, resilient data solutions across Azure, AWS, and Databricks, ensuring enterprise‑grade performance and reliability.
  • Lead complex engineering workstreams - unblocking teams, guiding technical decisions, and ensuring high‑quality delivery outcomes.
  • Design and govern enterprise data models (conceptual, logical, and physical), including lakehouse zone strategies and schema evolution approaches.
  • Champion Databricks as the core data platform, leveraging lakehouse architecture, Unity Catalog, Delta Live Tables, Structured Streaming, and MLflow.
  • Drive a consistent multi‑cloud Databricks strategy, enabling scalable and interoperable data platforms across environments.
  • Optimise performance across pipelines, compute, storage, and orchestration frameworks, embedding cost‑efficiency and scalability.
  • Define and implement CI/CD for data platforms, supporting automated testing, secure deployments, and multi‑environment promotion.
  • Embed robust data governance, security, and lifecycle controls, ensuring compliance through classification, encryption, access management, and lineage frameworks.
  • Act as a senior SME for troubleshooting complex issues, leading root‑cause analysis and long‑term remediation of data quality and platform challenges.
  • Lead, mentor, and grow a community of Data Engineers, driving technical excellence through coaching, reviews, and knowledge sharing.
  • Operate effectively across a matrix organisation, influencing architecture, cyber, product, and senior stakeholders.
  • Communicate complex technical concepts clearly, articulating trade‑offs, risks, and strategic decisions to both technical and non‑technical audiences.

Skills

PySpark
SQL
Data modelling
Leadership
Communication
Stakeholder management
Mentoring
CI/CD

Education

Azure Data Engineer Associate
AWS Data Analytics Specialty
Databricks Data Engineer (Associate/Professional)

Tools

Databricks
Azure
AWS
Unity Catalog
Delta Lake
Synapse
Kinesis

Job description

Serco is seeking a Lead Data Engineer to act as the technical authority for data engineering across a multi-cloud environment (Azure, AWS and Databricks). The role blends hands-on design and delivery (about 80%) with leadership of engineers, stakeholder influence and shaping data engineering practices.

You will champion Databricks as the core platform, govern data models, and drive cost-efficient, scalable solutions while collaborating across a matrix organisation to deliver enterprise-grade

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer - Multi-Cloud Databricks & Data Platform
Lead Data Engineer - Multi-Cloud Databricks & Data Platform

Serco Group Plc • Metropolitan Borough of Solihull

On-site
GBP 90,000 - 130,000
Lead Data Engineer - Multi-Cloud Databricks & Data Platform
Lead Data Engineer - Multi-Cloud Databricks & Data Platform

Serco Group Plc • Manor Park

On-site
GBP 90,000 - 120,000
Lead Data Engineer: Multi-Cloud Databricks Platform
Lead Data Engineer: Multi-Cloud Databricks Platform

Serco Group Plc • Metropolitan Borough of Solihull

On-site
GBP 90,000 - 130,000
Lead Data Engineer: Multi-Cloud Databricks Platform
Lead Data Engineer: Multi-Cloud Databricks Platform

Serco Group Plc • Manor Park

On-site
GBP 90,000 - 120,000
Lead Data Engineer - Hybrid, Multi-Cloud & Databricks Expert
Lead Data Engineer - Hybrid, Multi-Cloud & Databricks Expert

Serco Plc • Metropolitan Borough of Solihull, Greater London, Hook

Hybrid
GBP 75,000 - 85,000
Company car
5% on target bonus scheme
Private medical insurance
+2
Senior Data Engineer & Databricks Architect, Multi-Cloud
Senior Data Engineer & Databricks Architect, Multi-Cloud

Serco Group Plc • Metropolitan Borough of Solihull

On-site
GBP 90,000 - 120,000
Senior Data Engineer & Databricks Architect (Multi-Cloud)
Senior Data Engineer & Databricks Architect (Multi-Cloud)

Serco Limited • Metropolitan Borough of Solihull, Greater London, Hook

Hybrid
GBP 75,000 - 85,000
Company car
Bonus scheme
Private medical insurance
+5
Senior Data Engineer & Databricks Architect (Multi-Cloud)
Senior Data Engineer & Databricks Architect (Multi-Cloud)

Serco Group Plc • Manor Park

On-site
GBP 90,000 - 130,000
Senior Data Engineer & Databricks Architect (Multi-Cloud)
Senior Data Engineer & Databricks Architect (Multi-Cloud)

Serco Group Plc • Metropolitan Borough of Solihull

On-site
GBP 90,000 - 120,000
Senior Data Engineer & Databricks Architect (Multi-Cloud)
Senior Data Engineer & Databricks Architect (Multi-Cloud)

Serco Group Plc • Manor Park

On-site
GBP 90,000 - 130,000