Azure Databricks Data Engineer: Pipelines & Governance

Somerset Bridge; West of England; Hybrid; GBP 53,000 - 67,000; Full time

Benefits offered by this job

Hybrid working

25 days annual leave

Discretionary annual bonus

Pension scheme

Flexible working

Healthcare Cash Plan

Cycle to Work Scheme

Employee discounts

Job summary

Somerset Bridge Group is transforming its data platform to a modern Azure-based data ecosystem. As a Data Engineer you will design, build and optimise scalable data pipelines on Databricks and Delta Lake, supporting Pricing, Underwriting and analytics.

You will collaborate with Data Architects and Analysts to ensure data quality, governance and real-time/batch processing, delivering secure, high-performance data solutions across the organisation.

Qualifications

• Strong experience building data pipelines using Azure and Databricks.
• Proficiency in SQL and Python for data transformation.
• Experience with Delta Lake, lakehouse architectures and data modelling.
• Knowledge of data governance, security controls and regulatory compliance.
• Experience delivering scalable, high-quality data solutions for analytics, reporting and ML.

Responsibilities

• Design, build and maintain scalable data pipelines with Azure Databricks and Delta Lake.
• Develop and optimise cloud data solutions within Azure for performance and cost efficiency.
• Create robust data models and lakehouse architectures for analytics and ML.
• Automate and monitor data workflows with Databricks and Azure tools.
• Ensure data quality, governance and security across platforms.
• Collaborate with Pricing, Underwriting, Data Science and Tech teams.
• Implement data security and governance including access controls and data lineage.
• Drive continuous improvement with new Azure/Databricks technologies.

Skills

Azure Data Factory

Databricks

Delta Lake

SQL

Python / PySpark

Data Warehousing

Data Governance

CI/CD

Azure DevOps

Security & Compliance

Tools

Power BI

Databricks Workflows

Delta Live Tables

Azure Monitor

Unity Catalog

Purview

Job description

Somerset Bridge Group is transforming its data platform to a modern Azure-based data ecosystem. As a Data Engineer you will design, build and optimise scalable data pipelines on Databricks and Delta Lake, supporting Pricing, Underwriting and analytics.

You will collaborate with Data Architects and Analysts to ensure data quality, governance and real-time/batch processing, delivering secure, high-performance data solutions across the organisation.