Senior Data Engineer - Databricks & Azure, Flexible Hybrid

Howden Group; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Hybrid working

Employee ownership

Job summary

Howden Group is hiring a Senior Data Engineer to own metadata ingestion and orchestration, building scalable data pipelines with Databricks and Azure services. You will collaborate with stakeholders and junior engineers, balancing multiple deliverables while delivering robust data solutions to enable business insights and efficiencies.

The role emphasises autonomy, design decisions, and governance, with hybrid working and a culture that values ownership and collaboration.

Qualifications

• 4–6 years of experience in data engineering with a focus on Databricks (SQL & PySpark).
• Hands-on experience with Azure data services (ADF, ADLS, Functions).
• Proven track record designing metadata-driven data pipelines.
• Strong orchestration and data workflow automation knowledge (Airflow, Spark).
• Experience implementing CI/CD practices for data engineering.
• Experience contributing to technical design discussions and architecture.
• Good understanding of data governance, security and compliance.

Responsibilities

• Own metadata ingestion frameworks and orchestration processes.
• Design scalable, reusable metadata-driven services to optimise data pipelines.
• Build data solutions using Databricks and Azure services (ADF, ADLS, Functions).
• Promote CI/CD and DevOps best practices for data engineering.
• Collaborate with stakeholders, data scientists and analysts to translate needs.
• Enhance data pipelines for performance, reliability and scalability.
• Support data governance and security best practices across the team.
• Mentor junior engineers and review code to share best practices.
• Escalate complex data issues with strong problem solving and communication.

Skills

Databricks (SQL & PySpark)

Metadata ingestion

Orchestration & workflow automation

CI/CD for data engineering

Data governance & security

Stakeholder collaboration

Tools

Databricks

Azure Data Factory

Azure Data Lake Storage

Azure Functions

Airflow

DBT

Spark Declarative Pipelines

Job description

Howden Group is hiring a Senior Data Engineer to own metadata ingestion and orchestration, building scalable data pipelines with Databricks and Azure services. You will collaborate with stakeholders and junior engineers, balancing multiple deliverables while delivering robust data solutions to enable business insights and efficiencies.

The role emphasises autonomy, design decisions, and governance, with hybrid working and a culture that values ownership and collaboration.