Lead Data Engineer: Azure Databricks & Metadata Pipelines

Howden Group; England; Hybrid; GBP 90,000 - 140,000; Full time

Job summary

Howden Group is expanding its internal data engineering capability and seeks a Senior Data Engineer to lead high-impact initiatives on Databricks and Azure data services.

The role focuses on designing metadata-driven data pipelines, governance, and scalable solutions. You will mentor engineers, collaborate with stakeholders, and ensure robust, efficient data architectures across the Azure ecosystem.

Qualifications

• 8+ years in data engineering with Databricks (SQL & PySpark).
• Strong Azure data services experience (ADF, ADLS, Functions).
• Proven track record in metadata-driven data pipelines.
• Deep expertise in orchestration and data workflow automation (Airflow, DBT).

Responsibilities

• Lead metadata ingestion frameworks and orchestration processes.
• Design scalable, reusable metadata-driven services for pipelines.
• Architect and maintain Databricks/Azure data solutions.
• Establish CI/CD and DevOps practices for data.
• Collaborate with stakeholders to translate data needs.
• Optimize pipelines for performance and reliability.
• Promote data governance and security best practices.
• Lead ETL design and maintenance.
• Mentor junior engineers and share best practices.

Skills

Databricks SQL PySpark

Azure data services

CI/CD for data engineering

Data governance & security

Tools

Airflow

DBT

Terraform

Bicep

ARM templates

Job description

Howden Group is expanding its internal data engineering capability and seeks a Senior Data Engineer to lead high-impact initiatives on Databricks and Azure data services.

The role focuses on designing metadata-driven data pipelines, governance, and scalable solutions. You will mentor engineers, collaborate with stakeholders, and ensure robust, efficient data architectures across the Azure ecosystem.