Databricks Tech Lead - Azure Data Engineering (Contract) in London

Energy Jobline ZR; City Of London; On-site; GBP 70,000 - 90,000; Full time

Job summary

A leading energy sector company is seeking a Databricks Tech Lead for their Azure Data Engineering project. You will focus on building and scaling a data lake on Azure, managing streaming pipelines, and collaborating with stakeholders to optimize cloud resources. The ideal candidate has a strong background in Databricks, Delta Live Tables, and cost management in cloud environments. This role offers the opportunity to contribute to a large-scale digital transformation initiative.

Qualifications

• Strong background in Databricks and Azure Data Engineering.
• Experience with real-time streaming and batch jobs.
• Ability to troubleshoot and optimize performance in high-volume streaming systems.

Responsibilities

• Design and develop streaming and batch data pipelines in Databricks.
• Manage structured streaming for low-latency data delivery.
• Collaborate with architects and stakeholders to build a unified Azure-based lakehouse.
• Integrate data from on-prem and cloud sources.

Skills

Databricks

Azure Data Engineering

Streaming pipelines

Delta Live Tables

Qlik Replicate

Cloud cost management

Tools

Databricks

Delta Live Tables

Azure

Qlik Replicate (CDC)

Job description

Job Description

Databricks Tech Lead – Azure Data Engineering (Contract)

We are looking for a highly technical, hands‑on Databricks Tech Lead to join a large‑scale digital transformation programme within the Energy Trading space.

This role will focus on building, optimising, and scaling a data lake on Azure with particular emphasis on streaming pipelines, Delta Live Tables (DLT), and cost‑efficient architecture.

Role:

• Designing and developing streaming and batch data pipelines in Databricks (including DLT).
• Managing structured streaming in continuous mode, ensuring low‑latency data delivery.
• Collaborating with architects, consultants, and business stakeholders to build a unified Azure‑based lakehouse.
• Troubleshooting and optimising performance bottlenecks in high‑volume streaming systems.
• Applying best practices for cloud cost management and efficient Databricks resource usage.
• Integrating data from on‑prem and cloud sources using tools like Qlik Replicate (CDC).
• Supporting future phases involving data science and machine learning enablement.

Tech Environment

• Platform: Azure
• Core Tools: Databricks, Delta Live Tables, Qlik Replicate (CDC)
• Architecture: Hybrid (on‑prem + cloud)
• Data Modes: Real‑time streaming & batch jobs

If you are a Tech Lead with a strong background in Databricks, Delta Live Tables and Qlik Replicate, please apply with your updated CV.

If you are interested in applying for this job please press the Apply Button and follow the application process. Energy Jobline wishes you the very best of luck in your next career move.