Databricks Data Engineer

GIS Recruiter Ltd

City Of London

On-site

GBP 90,000 - 130,000

Full time

2 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

GIS Recruiter Ltd is seeking an experienced Senior Databricks Data Engineer to design and implement enterprise-scale data platforms using Azure Databricks for UK public sector clients. You will deliver complex ETL/ELT pipelines, Delta Lake solutions and Lakehouse architectures with strong governance.

Ideal candidates have 5+ years hands-on Databricks experience, expert PySpark and Spark SQL skills, and familiarity with Medallion Architecture, ADF integration, and data security standards.

Qualifications

  • 5+ years hands-on Databricks experience in enterprise data platforms.
  • Expert-level Azure Databricks and PySpark proficiency required.
  • Strong SQL skills and data modeling capabilities.
  • Experience with Delta Lake, Delta Live Tables and Lakehouse concepts.
  • Familiarity with Medallion Architecture and governance controls.
  • CI/CD tooling experience and automated deployments.
  • Public sector or government data security knowledge is a plus.

Responsibilities

  • Design and develop scalable Databricks data pipelines.
  • Build enterprise-grade Lakehouse platforms compliant with security standards.
  • Optimize Spark workloads and manage Databricks clusters.
  • Implement data quality, governance and validation frameworks.
  • Collaborate with architects, analysts and stakeholders across teams.
  • Support production deployments and troubleshoot data issues.
  • Ensure solutions align with UK public sector requirements.

Skills

Databricks
Azure Databricks
PySpark
Spark SQL
Python
ETL/ELT pipelines
Delta Lake
Delta Live Tables
Lakehouse architecture
Medallion Architecture
Spark performance tuning
Unity Catalog & data governance
CDC & SCD approaches
SQL development
Azure Data Factory integration
ADLS Gen2
CI/CD (Git, Azure DevOps)
Data quality frameworks

Job description

We are seeking an experienced Senior Databricks Data Engineer with extensive hands‑on experience designing, developing and implementing enterprise‑scale data engineering solutions using Azure Databricks. The successful candidate will have strong experience delivering complex data platforms within UK Public Sector environments, working with large datasets, sensitive information and secure cloud‑based data architectures.

Essential Technical Skills
  • 5+ years of hands‑on experience with Databricks.
  • Expert‑level knowledge of Azure Databricks.
  • Advanced PySpark, Spark SQL and Python programming.
  • Strong experience developing complex ETL/ELT pipelines.
  • Deep understanding of Delta Lake, Delta Tables and Delta Live Tables / Lakeflow Declarative Pipelines.
  • Experience implementing Medallion Architecture (Bronze, Silver and Gold).
  • Strong knowledge of Apache Spark architecture, optimisation and performance tuning.
  • Hands‑on experience with Unity Catalog, access controls and data governance.
  • Experience implementing incremental data loads, CDC and SCD Type 1/Type 2.
  • Strong SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks.
  • Knowledge of Azure Data Lake Storage Gen2.
  • Experience with CI/CD pipelines, Git, Azure DevOps and automated deployments.
  • Experience implementing data quality, validation and reconciliation frameworks.
Public Sector Experience
  • Previous experience delivering data engineering projects within UK Government, Central Government, Local Government, NHS, DWP, HMRC, Home Office or other public sector organisations.
  • Understanding of public sector data governance, security and information management.
  • Experience working with sensitive and confidential datasets.
  • Knowledge of UK GDPR, Data Protection Act 2018 and relevant government data security standards.
  • Experience working within Agile delivery teams and government digital transformation programmes.
Security Clearance Requirements
  • Be eligible and willing to undergo SC clearance.
  • Previous experience working on UK Government or Central Government projects is highly desirable.
  • Must be willing to undergo the required UK Government security vetting and background checks.
Responsibilities
  • Design and develop scalable Databricks data pipelines.
  • Build and maintain enterprise‑grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement robust data quality and governance controls.
  • Collaborate with architects, analysts and business stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance requirements
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Solution Architect
Databricks Solution Architect

GIS Recruiter Ltd • City Of London

On-site
GBP 120,000 - 165,000
Senior Databricks Data Engineer for Public Sector Platform
Senior Databricks Data Engineer for Public Sector Platform

GIS Recruiter Ltd • City Of London

On-site
GBP 90,000 - 130,000
Senior Data Engineer
Senior Data Engineer

Ceox • United Kingdom

Remote
GBP 83,000 - 131,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

ISx4 Group • Belfast City District

Hybrid
GBP 70,000 - 100,000
Databricks Platform Engineer
Databricks Platform Engineer

Sagacity Solutions Ltd. • Greater London

Hybrid
GBP 85,000 - 120,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Tenth Revolution Group • United Kingdom

Remote
GBP 90,000 - 130,000
Senior Data Engineer (Remote)
Senior Data Engineer (Remote)

Ceox Services Ltd • United Kingdom

Remote
GBP 92,000 - 129,000
Data Architect
Data Architect

Brio Digital • Greater London

Remote
GBP 112,000 - 147,000
Data Architect
Data Architect

Brio Digital • City Of London

Remote
GBP 112,000 - 147,000
Data Architect
Data Architect

Brio Digital • United Kingdom

Remote
GBP 112,000 - 147,000