Senior Data Engineer - Databricks & Azure, Hybrid Role

Howden Group Services Limited; City Of London; On-site; GBP 80,000 - 120,000; Full time

Job summary

Howden Group Services is expanding its internal data engineering capability and seeks a Senior Data Engineer with Databricks and Azure data services expertise to design scalable, metadata-driven data solutions. You will own ingestion frameworks, collaborate with stakeholders, and mentor junior engineers while delivering robust data pipelines.

The role emphasizes ownership of complex problems, autonomy, and driving best practices in CI/CD and data governance within a global insurance context.

Qualifications

• 4–6 years of data engineering experience focused on Databricks (SQL & PySpark).
• Hands-on Azure data services experience with ADF, ADLS, and Azure Functions.
• Experience designing metadata-driven data pipelines.
• Understanding of orchestration and data workflow automation (Airflow, DBT, Spark Declarative Pipelines).
• Knowledge of CI/CD practices for data engineering.
• Experience shaping technical design discussions and architectures.
• Awareness of data governance, security and compliance.
• Experience with insurance data (preferred).

Responsibilities

• Take ownership of metadata ingestion frameworks and orchestration processes.
• Design and implement scalable metadata-driven services to optimise data pipelines.
• Build and maintain robust data solutions using Databricks and Azure services (ADF, ADLS, Azure Functions).
• Apply and embed best practices for data engineering, CI/CD and DevOps for data.
• Collaborate with stakeholders, data scientists, and analysts to translate data needs into engineering solutions.
• Optimise and enhance data pipelines for performance, reliability, and scalability.
• Support data governance and security best practices across the team.
• Design, build and maintain ETL processes and analytical data products with minimal oversight.
• Provide technical guidance and mentoring to junior engineers; review code and share best practices.

Skills

Databricks (SQL & PySpark)

CI/CD for data engineering

Data governance & security

Tools

Azure Data Factory

Azure Data Lake Storage

Azure Functions

Autoloader

Spark Declarative Pipelines

Airflow

DBT

Job description

Howden Group Services is expanding its internal data engineering capability and seeks a Senior Data Engineer with Databricks and Azure data services expertise to design scalable, metadata-driven data solutions. You will own ingestion frameworks, collaborate with stakeholders, and mentor junior engineers while delivering robust data pipelines.

The role emphasizes ownership of complex problems, autonomy, and driving best practices in CI/CD and data governance within a global insurance context.