Lead Data Engineer — Databricks & Azure Solutions Architect

Howden Group; Greater London; Hybrid; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Hybrid working

Job summary

Howden Group is seeking a Senior Data Engineer to lead complex data initiatives using Databricks and Azure data services. The role focuses on metadata ingestion, orchestration, and scalable data pipelines to enable business insights.

The candidate will design robust data solutions, mentor junior engineers, and collaborate with stakeholders to drive best practices and governance across data platforms.

Qualifications

• 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark).
• Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions.
• Proven track record of designing and implementing metadata-driven data pipelines.
• Deep expertise in orchestration and data workflow automation e.g. Airflow, DBT.
• Strong understanding of CI/CD practices for data engineering.
• Experience with infrastructure as code (Terraform, Bicep, or ARM templates).
• Solid development and coding standards, with experience in software engineering best practices.
• Experience leading technical design discussions and driving architectural decisions.
• Strong stakeholder management skills and ability to translate business requirements into data solutions.
• Knowledge of data governance, security, and compliance best practices.

Responsibilities

• Lead metadata ingestion frameworks and orchestration processes.
• Design scalable, metadata-driven services to optimise data pipelines.
• Architect and maintain data solutions using Databricks and Azure services (ADF, ADLS, Azure Functions).
• Establish CI/CD pipelines and DevOps for data engineering.
• Collaborate with stakeholders, data scientists, and analysts to translate data needs into solutions.
• Optimise existing data pipelines for performance and scalability.
• Drive data governance and security best practices.
• Lead ETL design and mentor junior engineers.

Skills

Databricks (SQL & PySpark)

Azure data services

Airflow

DBT

CI/CD for data

Terraform

Bicep/ARM

Stakeholder management

Data governance

Job description

Howden Group is seeking a Senior Data Engineer to lead complex data initiatives using Databricks and Azure data services. The role focuses on metadata ingestion, orchestration, and scalable data pipelines to enable business insights.

The candidate will design robust data solutions, mentor junior engineers, and collaborate with stakeholders to drive best practices and governance across data platforms.