Senior Data Engineer: Azure/Databricks Tech Lead

Hyperiongrp

Greater London

Hybrid

GBP 85,000 - 120,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Hybrid working
Reasonable accommodations

Job summary

Howden Group Services is seeking a Senior Data Engineer to own high-impact data initiatives, focused on Databricks and Azure data services. You will design scalable metadata-driven data pipelines and collaborate with data scientists and analysts while mentoring junior engineers.

This role offers autonomy, exposure to a modern Azure-based data ecosystem, and opportunities to influence architecture and governance within a global insurance group.

Qualifications

  • 4–6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).
  • Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions.
  • Proven experience designing metadata-driven data pipelines.
  • Strong understanding of orchestration and data workflow automation (Airflow, DBT, Spark Declarative Pipelines).
  • Strong CI/CD practices for data engineering.
  • Experience contributing to technical design discussions and architectural decisions.
  • Good understanding of data governance, security, and compliance best practices.
  • Experience working with insurance data (preferred).

Responsibilities

  • Take ownership of the development of metadata ingestion frameworks and orchestration processes.
  • Design and implement scalable and reusable metadata-driven services to optimise data pipelines.
  • Build and maintain robust data solutions using Databricks and Azure services (such as Autoloader, Spark Declarative Pipelines, Azure Data Factory (ADF), Azure Functions, and Azure Data Lake Storage (ADLS)).
  • Apply and help embed best practices for data engineering, CI/CD pipelines, and DevOps for data.
  • Collaborate with business stakeholders, data scientists, and analysts to understand data needs and translate them into efficient engineering solutions.
  • Optimise and enhance existing data pipelines for performance, reliability, and scalability.
  • Support the adoption of data governance and security best practices across the team.
  • Design, build and maintain ETL processes and analytical data products with minimal oversight.
  • Provide technical guidance and informal mentoring to junior engineers, reviewing code and sharing best practices.
  • Act as a point of escalation for complex, business‑critical data issues, demonstrating strong problem solving, communication, and stakeholder management skills.

Skills

Databricks
Metadata-driven pipelines
CI/CD for data
Data governance
Orchestration concepts

Tools

Airflow
DBT
Azure Data Factory
Azure Data Lake Storage
Azure Functions

Job description

Howden Group Services is seeking a Senior Data Engineer to own high-impact data initiatives, focused on Databricks and Azure data services. You will design scalable metadata-driven data pipelines and collaborate with data scientists and analysts while mentoring junior engineers.

This role offers autonomy, exposure to a modern Azure-based data ecosystem, and opportunities to influence architecture and governance within a global insurance group.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer - Databricks & Azure, Hybrid Role
Senior Data Engineer - Databricks & Azure, Hybrid Role

Howden Group Services Limited • City Of London

On-site
GBP 80,000 - 120,000
Senior Data Engineer: Databricks, Azure & Metadata-Driven
Senior Data Engineer: Databricks, Azure & Metadata-Driven

Howden Group Holdings • City Of London

Hybrid
GBP 90,000 - 120,000
Senior Data Engineer - Databricks & Azure Lead
Senior Data Engineer - Databricks & Azure Lead

Howden • Greater London

Hybrid
GBP 70,000 - 115,000
Hybrid working
Collaborative culture
Senior Data Engineer — Databricks on Azure (Hybrid/Flexible)
Senior Data Engineer — Databricks on Azure (Hybrid/Flexible)

Howden Group • England

Hybrid
GBP 75,000 - 110,000
Senior Data Engineer | Databricks & Azure | Hybrid
Senior Data Engineer | Databricks & Azure | Hybrid

TigerRisk Partners, LLC. • Greater London

Hybrid
GBP 70,000 - 110,000
Hybrid working
Lead Data Engineer: Azure Databricks & Metadata Pipelines
Lead Data Engineer: Azure Databricks & Metadata Pipelines

Howden Group Holdings • England

Hybrid
GBP 90,000 - 140,000
Lead Data Engineer
Lead Data Engineer

Howden Group Holdings • England

Hybrid
GBP 90,000 - 140,000
Senior Data Engineer
Senior Data Engineer

Hyperiongrp • Greater London

Hybrid
GBP 85,000 - 120,000
Hybrid working
Reasonable accommodations
Senior Data Platform Engineer - Azure Databricks Lakehouse
Senior Data Platform Engineer - Azure Databricks Lakehouse

Source Group International • Greater London

On-site
GBP 90,000 - 125,000
Senior Data Engineer
Senior Data Engineer

Howden Group Holdings • England

Hybrid
GBP 75,000 - 110,000