Azure Data Platform Engineer: Databricks & Delta Lake

Somerset Bridge; Newcastle upon Tyne; Hybrid; GBP 53,000 - 67,000; Full time

Benefits offered by this job

Hybrid working - 2 days in the office

25 days annual leave

Discretionary annual bonus

Pension scheme - 5% employee, 6% 
empr

Job summary

Somerset Bridge seeks a Data Engineer to design, build and optimise data pipelines across Pricing, Underwriting and Claims. You’ll work on a modern Azure-based data platform, consolidating capabilities on Databricks, Delta Lake and Unity Catalog to support analytics and AI initiatives.

You’ll collaborate with Data Architects and Analysts, delivering scalable, secure data solutions in a hybrid setup with both real-time and batch processing.

Qualifications

• Strong experience building data pipelines with Azure Data Factory, Databricks and Delta Lake.
• Advanced SQL and Python (PySpark) skills for data transformation and automation.
• Hands-on with Delta Lake, Spark, Databricks Workflows and Structured Streaming.
• Knowledge of data warehousing, lakehouse architectures and dimensional modelling.
• Experience delivering scalable data solutions for analytics, reporting and ML.
• Familiar with real-time data processing, APIs and streaming tech like Kafka.

Responsibilities

• Design, build and maintain scalable data pipelines for real-time and batch processing.
• Develop and optimise cloud-based data solutions within Azure for performance and cost.
• Create robust data models and lakehouse architectures for analytics and ML.
• Automate and monitor data workflows with Databricks and Azure tools.
• Ensure data quality, governance, security controls and regulatory compliance.
• Collaborate with Pricing, Underwriting, Data Science and Tech teams.
• Implement data security, access controls and data lineage.
• Drive continuous improvement using new Azure/Databricks technologies.

Skills

Azure Databricks

Azure Data Factory

Delta Lake

PySpark

SQL

Python

Data Governance

CI/CD

Azure DevOps

Security & Compliance

Stakeholder Management

Tools

Databricks Workflows

Delta Live Tables

Azure Monitor

Unity Catalog

Purview

Kafka

GitHub Actions

Job description

Somerset Bridge seeks a Data Engineer to design, build and optimise data pipelines across Pricing, Underwriting and Claims. You’ll work on a modern Azure-based data platform, consolidating capabilities on Databricks, Delta Lake and Unity Catalog to support analytics and AI initiatives.

You’ll collaborate with Data Architects and Analysts, delivering scalable, secure data solutions in a hybrid setup with both real-time and batch processing.