Data Engineer: Databricks & Fabric Lakehouse Pipelines

Leonard Curtis; Whitefield; On-site; GBP 65,000 - 90,000; Full time

Benefits offered by this job

25 days holiday

Birthday leave

Giving back days

Enhanced family friendly policies

Job summary

Leonard Curtis is seeking a Data Engineer to build, maintain, and optimise robust data pipelines. This role delivers clean, production-ready data for enterprise analytics using Microsoft Fabric tools and the Databricks Lakehouse Platform.

You will develop ETL/ELT workflows, implement automated data quality checks, and support real-time Power BI reporting needs while adhering to governance policies and industry best practices.

Qualifications

• Strong proficiency in PySpark, Databricks SQL, T-SQL and Delta Lake formats.
• Hands-on experience building production pipelines in Databricks and Fabric.
• Knowledge of Parquet, JSON, and CSV formats.

Responsibilities

• Construct scalable data ingestion pipelines using Fabric Data Factory, Dataflow Gen2 and Databricks Workflows.
• Implement Delta Live Tables to automate batch and real-time processing.
• Build and maintain silver and gold layers in shared storage with Medallion Architecture.
• Write optimised transformations in Fabric Notebooks and Databricks clusters using PySpark and SQL.
• Configure data access, masking and governance in Databricks Unity Catalog.
• Optimise Lakehouse performance with indexing and clustering techniques.
• Deploy pipelines using CI/CD, Git and Databricks Asset Bundles.

Skills

PySpark

Databricks SQL

T-SQL

Delta Lake

Tools

Databricks Workspaces

Fabric Data Factory

Dataflow Gen2

Databricks Workflows

Fabric Notebooks

Databricks Clusters

Unity Catalog

Job description

Leonard Curtis is seeking a Data Engineer to build, maintain, and optimise robust data pipelines. This role delivers clean, production-ready data for enterprise analytics using Microsoft Fabric tools and the Databricks Lakehouse Platform.

You will develop ETL/ELT workflows, implement automated data quality checks, and support real-time Power BI reporting needs while adhering to governance policies and industry best practices.