Databricks Data Engineer

GIS Recruiter Ltd

Greater London

On-site

GBP 90,000 - 120,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

GIS Recruiter Ltd seeks an experienced Senior Databricks Data Engineer to design and implement enterprise-scale data platforms in Azure Databricks for UK public sector environments. You will work with large, sensitive datasets and secure cloud architectures, delivering Delta Lake and lakehouse solutions with robust governance.

The role emphasizes PySpark, SQL optimisation, CI/CD pipelines, and collaboration with architects, analysts and government stakeholders to meet strict security and

Qualifications

  • 5+ years hands-on Databricks experience.
  • Expert Azure Databricks knowledge.
  • Strong PySpark, Spark SQL and Python.
  • Experience with Delta Lake and related Lakehouse architectures.
  • Experience implementing Medallion Architecture (Bronze, Silver, Gold).
  • Knowledge of Spark architecture, optimisation and performance tuning.
  • Unity Catalog and data governance experience.
  • Incremental data loads, CDC and SCD Type 1/2 experience.
  • SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks.

Responsibilities

  • Design scalable Databricks data pipelines.
  • Build and maintain enterprise-grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement data quality and governance controls.
  • Collaborate with architects, analysts and stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance requirements

Skills

Databricks
Azure Databricks
PySpark
Python
ETL Pipelines
Delta Lake
Lakehouse
Spark Tuning
Data Governance
SQL
CI/CD
Git
Azure DevOps

Tools

Azure Data Factory
Git
Azure DevOps

Job description

We are seeking an experienced Senior Databricks Data Engineer with extensive hands-on experience designing, developing and implementing enterprise-scale data engineering solutions using Azure Databricks. The successful candidate will have strong experience delivering complex data platforms within UK Public Sector environments, working with large datasets, sensitive information and secure cloud-based data architectures.

Essential Technical Skills
  • 5+ years of hands-on experience with Databricks.
  • Expert-level knowledge of Azure Databricks.
  • Advanced PySpark, Spark SQL and Python programming.
  • Strong experience developing complex ETL/ELT pipelines.
  • Deep understanding of Delta Lake, Delta Tables and Delta Live Tables / Lakeflow Declarative Pipelines.
  • Experience implementing Medallion Architecture (Bronze, Silver and Gold).
  • Strong knowledge of Apache Spark architecture, optimisation and performance tuning.
  • Hands-on experience with Unity Catalog, access controls and data governance.
  • Experience implementing incremental data loads, CDC and SCD Type 1/Type 2.
  • Strong SQL development and query optimisation skills.
  • Experience integrating Azure Data Factory with Databricks.
  • Knowledge of Azure Data Lake Storage Gen2.
  • Experience with CI/CD pipelines, Git, Azure DevOps and automated deployments.
  • Experience implementing data quality, validation and reconciliation frameworks.
Public Sector Experience
  • Previous experience delivering data engineering projects within UK Government, Central Government, Local Government, NHS, DWP, HMRC, Home Office or other public sector organisations.
  • Understanding of public sector data governance, security and information management.
  • Experience working with sensitive and confidential datasets.
  • Knowledge of UK GDPR, Data Protection Act 2018 and relevant government data security standards.
  • Experience working within Agile delivery teams and government digital transformation programmes.
Security Clearance Requirements
  • Be eligible and willing to undergo SC clearance.
  • Previous experience working on UK Government or Central Government projects is highly desirable.
  • Must be willing to undergo the required UK Government security vetting and background checks.
Responsibilities
  • Design and develop scalable Databricks data pipelines.
  • Build and maintain enterprise-grade Lakehouse platforms.
  • Optimise Spark workloads and Databricks clusters.
  • Implement robust data quality and governance controls.
  • Collaborate with architects, analysts and business stakeholders.
  • Support production deployments and troubleshoot complex data issues.
  • Ensure solutions meet public sector security and compliance requirements
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Data Engineer
Databricks Data Engineer

GIS Recruiter Ltd • City Of London

On-site
GBP 90,000 - 130,000
Databricks Solution Architect
Databricks Solution Architect

GIS Recruiter Ltd • City Of London

On-site
GBP 120,000 - 165,000
Databricks Solution Architect
Databricks Solution Architect

GIS Recruiter Ltd • Greater London

On-site
GBP 90,000 - 130,000
Senior Databricks Data Engineer for Public Sector Platform
Senior Databricks Data Engineer for Public Sector Platform

GIS Recruiter Ltd • City Of London

On-site
GBP 90,000 - 130,000
Senior Data Engineer
Senior Data Engineer

Ceox • United Kingdom

Remote
GBP 83,000 - 131,000
Senior Azure Databricks Data Engineer - Public Sector
Senior Azure Databricks Data Engineer - Public Sector

GIS Recruiter Ltd • Greater London

On-site
GBP 90,000 - 120,000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

ISx4 Group • Belfast City District

Hybrid
GBP 70,000 - 100,000
Databricks Platform Engineer
Databricks Platform Engineer

Sagacity Solutions Ltd. • Greater London

Hybrid
GBP 85,000 - 120,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Tenth Revolution Group • United Kingdom

Remote
GBP 90,000 - 130,000
Databricks Engineer
Databricks Engineer

Gazelle Global Consulting Limited • Greater London

Hybrid
GBP 70,000 - 95,000