Lead Data Engineer - AI Data Platform Lead

Xdesign

City of Edinburgh

Hybrid

GBP 90,000 - 130,000

Full time

9 days ago

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Benefits offered by this job

Hybrid and remote options

Job summary

CreateFuture in the UK is seeking a Lead Data Engineer to own the technical direction of a data engagement, leading a team and working with Microsoft Fabric, Python and PySpark across our data team. You’ll design data platform architecture, set standards for pipelines and CI/CD, and work closely with client stakeholders to ensure delivery is technically excellent and commercially sound.

You’ll also lead incident response, drive architectural decisions and build scalable data solutions across

Qualifications

We are looking for a Lead Data Engineer with hands-on experience in modern cloud data platforms and strong engineering depth.
Expertise in SQL, data modelling and architecture (Kimball, data vault, medallion/lakehouse).
Proficiency in Python and PySpark for batch/ELT data processing at scale.
Experience leading a team of data engineers and setting standards for pipeline design and delivery.
Experience owning CI/CD and operational practices for data workflows, ideally using Azure DevOps.
Experience working directly with client stakeholders and familiarity with the Azure ecosystem.

Responsibilities

Own the technical direction of a data engagement and set pipeline architecture standards.
Lead and direct a team of data engineers, reviewing designs and code.
Design and evolve data platform architecture on Microsoft Fabric, including pipelines and Azure OneLake.
Set CI/CD, testing and operational practices across the data engineering workflows using Azure DevOps.
Translate business and data requirements into a technical roadmap for clients.
Troubleshoot complex data pipeline issues and lead incident response when needed.