Senior Data Engineer: Metadata-Driven Azure Databricks

hyperiongrp; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Hybrid working

Reasonable adjustments

Job summary

Howden is expanding its data engineering capability and seeks a Senior Data Engineer to lead metadata-driven data initiatives. You will design scalable data pipelines using Databricks and Azure, mentor junior engineers, and collaborate with stakeholders to deliver robust data solutions for insights and operational efficiency.

The role requires strategic thinking, leadership in design discussions, and a commitment to best practices in CI/CD and DevOps for data across a modern Azure-based data

Qualifications

• 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).
• Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions.
• Proven track record designing metadata-driven data pipelines.
• Deep expertise in orchestration and data workflow automation (Airflow, DBT).
• Strong CI/CD practices for data engineering; IaC experience (Terraform, Bicep or ARM).
• Solid software engineering practices and ability to drive architectural decisions.
• Excellent stakeholder management and ability to translate business needs into data solutions.

Responsibilities

• Lead the development of metadata ingestion frameworks and orchestration processes.
• Design scalable, reusable metadata-driven services to optimise data pipelines.
• Architect and maintain data solutions using Databricks and Azure services (Autoloader, Spark Declarative Pipelines, ADF, ADLS, Azure Functions, ADLS).
• Establish and enforce best practices for data engineering, CI/CD pipelines, and DevOps for data.
• Collaborate with business stakeholders, data scientists, and analysts to translate needs into solutions.
• Optimize and enhance existing data pipelines for performance, reliability, and scalability.
• Drive adoption of data governance and security best practices.
• Lead the design and maintenance of ETL processes; mentor junior engineers.

Skills

Leadership

Mentoring

Stakeholder management

Strategic thinking

Tools

Databricks

Azure data services

Airflow

DBT

Terraform

Bicep

ARM templates

Job description

Howden is expanding its data engineering capability and seeks a Senior Data Engineer to lead metadata-driven data initiatives. You will design scalable data pipelines using Databricks and Azure, mentor junior engineers, and collaborate with stakeholders to deliver robust data solutions for insights and operational efficiency.

The role requires strategic thinking, leadership in design discussions, and a commitment to best practices in CI/CD and DevOps for data across a modern Azure-based data