Senior Data Engineer: Metadata-Driven on Azure & Databricks

Howden; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

Howden is seeking a Senior Data Engineer to lead complex data initiatives within a modern Azure-based data ecosystem. You will drive metadata-driven designs and scalable pipelines, partnering with stakeholders to deliver impactful insights.

You will mentor engineers, set best practices, and own architectural decisions, overseeing end-to-end data solutions that enhance business efficiency and governance.

Qualifications

• 8+ years of experience in data engineering with focus on Databricks (SQL & PySpark).
• Extensive experience with Azure data services (ADF, ADLS, Functions).
• Proven track record designing metadata-driven data pipelines.
• Deep expertise in orchestration and data workflow automation (Airflow, DBT).
• Strong CI/CD practices for data engineering; IaC experience preferred (Terraform/Bicep/ARM).
• Solid software engineering practices and architectural decision-making.
• Experience leading technical design discussions and stakeholder alignment.
• Knowledge of data governance, security, and compliance best practices.

Responsibilities

• Lead metadata ingestion frameworks and orchestration processes.
• Design scalable, metadata-driven services to optimize data pipelines.
• Architect and maintain data solutions using Databricks and Azure (ADF, ADLS, Functions, ADLS).
• Enforce data engineering CI/CD and DevOps practices.
• Collaborate with stakeholders, data scientists, and analysts to translate needs into solutions.
• Optimize pipelines for performance, reliability, and scalability.
• Drive data governance and security best practices.
• Lead design and maintenance of ETL processes.
• Mentor and support junior engineers, promoting continuous improvement.

Skills

Databricks expertise

Azure data services

Metadata-driven pipelines

Leadership & mentoring

CI/CD for data

Airflow

DBT

Terraform / IaC

Tools

Azure Data Factory

Azure Data Lake Storage

Autoloader

Spark Declarative Pipelines

Job description

Howden is seeking a Senior Data Engineer to lead complex data initiatives within a modern Azure-based data ecosystem. You will drive metadata-driven designs and scalable pipelines, partnering with stakeholders to deliver impactful insights.

You will mentor engineers, set best practices, and own architectural decisions, overseeing end-to-end data solutions that enhance business efficiency and governance.