Data Engineer - Delta Lake & Databricks Pipelines

Leonard Curtis Ltd; Manchester; Hybrid; GBP 70,000 - 110,000; Full time

Benefits offered by this job

27 days holiday allowance

Birthday leave

Giving back days (2)

Enhanced family friendly policies

Enhanced Maternity pay

Job summary

Leonard Curtis Ltd in Manchester seeks a Data Engineer to build, maintain and optimise robust data pipelines. You will develop ETL/ELT workflows using Microsoft Fabric tools and the Databricks Lakehouse Platform to deliver clean data for enterprise analytics.

You will work with PySpark, Delta Lake formats, and Databricks notebooks to implement scalable pipelines, data quality checks, and CI/CD deployment. This role offers a collaborative, professional environment within a market-leading firm.

Qualifications

• Proficiency in PySpark, Databricks SQL, and T-SQL.
• Experience handling Delta Lake formats ( Parquet/JSON/CSV ).
• Hands-on experience building production pipelines in Databricks and Fabric.
• Knowledge of orchestration tools, scheduling, and error handling in cloud environments.
• Familiarity with unit testing data pipelines and automated data quality checks.

Responsibilities

• Construct scalable data ingestion pipelines using Fabric Data Factory, Dataflow Gen2, and Databricks Workflows.
• Implement Delta Live Tables (DLT) to automate batch and real-time data processing.
• Build and maintain Medallion Architecture silver & gold layers across shared storage.
• Write optimized transformations in Fabric Notebooks and Databricks clusters using PySpark and SQL.
• Configure data access, masking, and governance policies within Databricks Unity Catalog.
• Optimize Lakehouse table performance using Z-Order indexing, liquid clustering, and file compaction.
• Deploy data pipelines using CI/CD frameworks, Git integration, and Databricks Asset Bundles (DABs).
• Prepare finalized gold layer datasets to support real-time Power BI Direct Lake reporting model

Skills

PySpark

Databricks SQL

T-SQL

Delta Lake formats

Unit testing data pipelines

Data quality checks

CI/CD pipelines

Tools

Databricks

Fabric Data Factory

Dataflow Gen2

Delta Live Tables

Databricks Unity Catalog

Z-Order indexing

Job description

Leonard Curtis Ltd in Manchester seeks a Data Engineer to build, maintain and optimise robust data pipelines. You will develop ETL/ELT workflows using Microsoft Fabric tools and the Databricks Lakehouse Platform to deliver clean data for enterprise analytics.

You will work with PySpark, Delta Lake formats, and Databricks notebooks to implement scalable pipelines, data quality checks, and CI/CD deployment. This role offers a collaborative, professional environment within a market-leading firm.