Data Engineer

Somerset Bridge; Newcastle upon Tyne; Hybrid; GBP 53,000 - 67,000; Full time

Benefits offered by this job

Hybrid working - 2 days in the office

25 days annual leave

Discretionary annual bonus

Pension scheme - 5% employee, 6% 
empr

Job summary

Somerset Bridge seeks a Data Engineer to design, build and optimise data pipelines across Pricing, Underwriting and Claims. You’ll work on a modern Azure-based data platform, consolidating capabilities on Databricks, Delta Lake and Unity Catalog to support analytics and AI initiatives.

You’ll collaborate with Data Architects and Analysts, delivering scalable, secure data solutions in a hybrid setup with both real-time and batch processing.

Qualifications

• Strong experience building data pipelines with Azure Data Factory, Databricks and Delta Lake.
• Advanced SQL and Python (PySpark) skills for data transformation and automation.
• Hands-on with Delta Lake, Spark, Databricks Workflows and Structured Streaming.
• Knowledge of data warehousing, lakehouse architectures and dimensional modelling.
• Experience delivering scalable data solutions for analytics, reporting and ML.
• Familiar with real-time data processing, APIs and streaming tech like Kafka.

Responsibilities

• Design, build and maintain scalable data pipelines for real-time and batch processing.
• Develop and optimise cloud-based data solutions within Azure for performance and cost.
• Create robust data models and lakehouse architectures for analytics and ML.
• Automate and monitor data workflows with Databricks and Azure tools.
• Ensure data quality, governance, security controls and regulatory compliance.
• Collaborate with Pricing, Underwriting, Data Science and Tech teams.
• Implement data security, access controls and data lineage.
• Drive continuous improvement using new Azure/Databricks technologies.

Skills

Azure Databricks

Azure Data Factory

Delta Lake

PySpark

SQL

Python

Data Governance

CI/CD

Azure DevOps

Security & Compliance

Stakeholder Management

Tools

Databricks Workflows

Delta Live Tables

Azure Monitor

Unity Catalog

Purview

Kafka

GitHub Actions

Job description

Data Engineer

Department: [SBSS] Enterprise Data Management

Employment Type: Permanent - Full Time

Location: Newcastle

Compensation: £53,000 - £67,000 / year

Description

At Somerset Bridge Group, we're transforming our data platform to create a modern, cloud-based data ecosystem that powers smarter decision-making across the business. As part of this journey, we're consolidating our data capabilities onto a single Azure Databricks platform, leveraging technologies including Azure Data Factory, Delta Lake, Unity Catalog and Power BI to unlock new insights and drive innovation.

Joining our Enterprise Data Team, you'll play a key role in designing, building and optimising scalable data solutions that support Pricing, Underwriting, Claims and wider business operations. Working with large and complex datasets, you'll develop robust data pipelines and platforms that enable analytics, reporting, machine learning and AI-driven initiatives across the organisation.

You'll be responsible for building and maintaining data pipelines using Azure Databricks, Azure Data Factory and Delta Lake, supporting the modernisation of legacy platforms while ensuring data quality, reliability and performance. The role will also involve implementing data governance and security controls, optimising cloud-based data infrastructure, and supporting both real-time and batch data processing.

Working closely with Data Architects, Analysts, Pricing teams and Technology colleagues, you'll help shape the future of SBG's enterprise data platform, delivering scalable, secure and efficient solutions that support strategic business objectives and regulatory requirements.

This is an exciting opportunity to join a growing Enterprise Data Team and make a significant impact on the development of a modern Azure-based data platform, helping to drive data-led innovation across the business.

What you'll be responsible for:

• Design, build and maintain scalable data pipelines using Azure Databricks, Azure Data Factory and Delta Lake to support both real-time and batch data processing.
• Develop and optimise cloud-based data solutions within Azure, ensuring high performance, scalability and cost efficiency.
• Create robust data models and lakehouse architectures to support analytics, reporting and machine learning initiatives.
• Automate and monitor data workflows using tools such as Databricks Workflows, Delta Live Tables and Azure Monitor.
• Ensure data quality, integrity and governance through effective controls, security standards and regulatory compliance.
• Collaborate with stakeholders across Pricing, Underwriting, Data Science and Technology to deliver data-driven business solutions.
• Implement data security and governance frameworks, including access controls, data lineage and sensitive data protection.
• Drive continuous improvement by adopting new technologies and best practices across the Azure and Databricks ecosystem.

What you’ll need:

• Strong experience building data pipelines using Azure Data Factory, Databricks and Azure Data Lake.
• Advanced SQL and Python (PySpark) skills for data transformation, automation and optimisation.
• Hands-on experience with Delta Lake, Spark, Databricks Workflows and Structured Streaming.
• Knowledge of data warehousing, lakehouse architectures and dimensional modelling.
• Experience delivering scalable, high-quality data solutions to support analytics, reporting and machine learning.
• Familiarity with real-time data processing, APIs and streaming technologies such as Kafka or Azure Event Hubs.
• Experience with CI/CD, Azure DevOps, GitHub Actions and Infrastructure as Code.
• Strong understanding of data governance, security and access controls, including Unity Catalog and Purview.
• Experience monitoring, optimising and troubleshooting cloud-based data platforms.
• Excellent problem-solving, communication and stakeholder management skills.
• Collaborative approach with the ability to work effectively across technical and business teams.
• Passion for innovation, continuous improvement and emerging data technologies.

Desirable

• Experience with Scala.
• Exposure to MLOps, feature stores and machine learning platforms.

Our Benefits

• Hybrid working - 2 days in the office and 3 days working from home
• 25 days annual leave, rising to 27 days over 2 years' service and 30 days after 5 years' service. Plus bank holidays!
• Discretionary annual bonus
• Pension scheme - 5% employee, 6% employer
• Flexible working - we will always consider applications for those who require less than the advertised hours
• Flexi-time
• Healthcare Cash Plan - claim cashback on a variety of everyday healthcare costs
• Electric vehicle - salary sacrifice scheme
• 100's of exclusive retailer discounts
• Professional wellbeing, health & fitness app - Wrkit
• Enhanced parental leave, including time off for IVF appointments
• Religious bank holidays - if you don't celebrate Christmas and Easter, you can use these annual leave days on other occasions throughout the year.
• Life Assurance - 4 times your salary
• 25% Car Insurance Discount
• 20% Travel Insurance Discount
• Cycle to Work Scheme
• Employee Referral Scheme
• Community support day