Lead Data and AI Engineer in Leeds, West Yorkshire

Energy Jobline CVL; Leeds; Hybrid; GBP 76,000 - 90,000; Full time

Job summary

A leading recruitment firm in the UK is seeking a Data Engineering Lead to architect and implement cloud-data solutions. This pivotal role involves defining the data engineering strategy, leading the development of data pipelines, and delivering secure and scalable platforms using Azure. Candidates should have extensive experience in cloud-based data engineering and relevant certifications. The position offers a salary up to £90,000 along with an excellent benefits package and flexible working arrangements.

Benefits

Generous annual leave

Pension

Flexible working arrangements

Qualifications

• Extensive experience in cloud-based data engineering.
• Expertise in Azure services, Databricks, Spark, Python, SQL.
• Proven track record of delivering complex data solutions.

Responsibilities

• Define and implement the data engineering strategy.
• Lead development of data ingestion and quality pipelines.
• Deliver scalable data platforms using Azure and Databricks.

Skills

Cloud-based data engineering

Data governance

Python

SQL

Azure services

DevOps practices

AI/ML frameworks

Education

Relevant certifications in data engineering

Advanced academic qualifications in cloud technologies

Tools

Databricks

Azure

GitHub Actions

Job description

Role Overview

Energy Jobline is the largest global Energy Job Board with over 7 million professionals. We are recruiting for a pivotal opportunity to join the Data and Innovation division of a large complex organisation leading the delivery of the Supervisory Analytics and Metrics (SAM) program.

As Data Engineering Lead, you will architect and implement cloud‑data solutions aligned with the organisation’s enterprise cloud strategy and SAM’s Target Operating Model.

Responsibilities

• Define and implement the data engineering strategy aligned with business and technology goals.
• Lead development of data ingestion, quality, and metadata pipelines powering SAM’s supervisory tools.
• Deliver scalable, secure, and production‑ready data platforms using Azure and Databricks.
• Collaborate across Technology and DAT to integrate SAM solutions into the organisation’s Enterprise Data Platform (EDP).
• Champion CI/CD, DevOps, data governance and federated development within the PRA’s Hub & Spoke model.
• Mentor and coach data engineers on Azure tooling, pipeline management, coding practices, and design principles.
• Work with data governance teams to maintain a comprehensive data catalogue and ensure compliance with security and privacy regulations.
• Contribute to Communities of Practice and support the cloud‑first strategy.

Qualifications

• Extensive experience in cloud‑based data engineering (preferably Databricks), with a strong background in modernization and large‑scale migration.
• Expertise in Azure services (API Manager, App Service), Databricks, Spark, Python, SQL, and AI/ML frameworks.
• Proven track record of leading technical teams and delivering complex data solutions in production.
• Strong understanding of data governance, security, and compliance in regulated environments.
• Proven experience designing and deploying cloud‑data architectures at scale.
• Proficiency in Python, SQL, PySpark.
• Demonstrated ability to build secure, scalable, cost‑efficient data solutions on Azure.
• Experience with data security and regulatory compliance tools (e.g. Microsoft Purview, Unity Catalog).
• Experience designing and implementing AI/ML‑driven solutions within data platforms.
• Relevant certifications (e.g. Databricks Engineer Professional, Azure Data Engineer Associate, Azure Solutions Architect).
• Advanced academic qualifications or industry recognition in data engineering and cloud technologies.
• Experience in DevOps practices using GitHub Actions and automated CI/CD pipelines.

Benefits & Compensation

Salary negotiable on experience up to £90,000 plus excellent benefits package including generous annual leave, pension and hybrid working.

Based in Leeds with flexible working arrangements.

How to Apply

If you are interested in this role, click ‘apply now’ to forward an up‑to‑date copy of your CV, or call us now.

Energy Jobline wishes you the very best of luck in your next career move.