Senior Data Engineer - Azure & Databricks Platform Expert

Clarion Events; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

25 days holiday

Private Medical Insurance

Pension Scheme

Health Cash Plan

Cycle to Work Scheme

Free on-site gym

Season Ticket Loan

Wellbeing Library

Mentoring Programme

End of year wellbeing shutdown

Job summary

Clarion Events is seeking a Senior Data Engineer to design, build, and scale robust data pipelines on the Azure Databricks platform. You will shape data models, governance, and orchestration across APIs, SQL sources, and SaaS feeds, enabling analytics and AI initiatives globally.

You will implement best-practice CI/CD for data, collaborate with analytics teams, and mentor junior engineers while maintaining strong security and production readiness. London-based role with strong engineering focus.

Qualifications

• 5+ years in data engineering across cloud-native environments.
• Expert knowledge of Databricks, Delta Lake, Unity Catalog, ADF, Synapse, Event Hub, and wider Azure Data Services.
• Highly proficient in Python and SQL, with experience building production-grade data pipelines.
• Strong understanding of Lakehouse design patterns, Medallion modelling, and performance optimisation.
• Hands-on experience with CI/CD for data (Azure DevOps, YAML pipelines, Terraform, Databricks CLI/Repos).
• Solid grasp of data governance, security principles, and operational best practices.

Responsibilities

• Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake.
• Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms).
• Implement strong data quality, lineage, and governance standards aligned to Medallion Architecture (Bronze/Silver/Gold).
• Build and optimise Lakehouse data models to support analytics, reporting, operational workloads, and downstream product teams.
• Develop reusable frameworks for ingestion, validation, schema evolution, and monitoring.
• Ensure all solutions follow Databricks best practices around Unity Catalog, cluster policies, permissions, and environment separation.
• Work with architects to evolve Databricks platform patterns, security, and performance optimisation.
• Implement monitoring, logging, and observability across pipelines and clusters.
• Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets.
• Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required.
• Partner with analysts and data scientists to ensure high-quality, well-modelled data for analytics and AI use cases.
• Assist with productionising ML pipelines where needed, ensuring they run reliably within Databricks.
• Support feature engineering and experiment environments as part of the wider platform.
• Work cross-functionally with product, marketing, operations, and technology teams to understand data needs and accelerate delivery.
• Mentor junior engineers and promote engineering best practices.
• Champion DevOps for data: CI/CD pipelines, testing automation, IaC, and environment management.

Skills

Python

SQL

Databricks

Delta Lake

Unity Catalog

Synapse

Azure Data Factory

Event Hub

Azure Data Services

CI/CD for data

Tools

Databricks

Delta Lake

Unity Catalog

ADF

Synapse

Event Hub

Azure

Job description

Clarion Events is seeking a Senior Data Engineer to design, build, and scale robust data pipelines on the Azure Databricks platform. You will shape data models, governance, and orchestration across APIs, SQL sources, and SaaS feeds, enabling analytics and AI initiatives globally.

You will implement best-practice CI/CD for data, collaborate with analytics teams, and mentor junior engineers while maintaining strong security and production readiness. London-based role with strong engineering focus.