Senior Data Engineer

Opus Recruitment Solutions; Greater London; Hybrid; GBP 83,025 - 101,475; Part time

Job summary

Opus Recruitment Solutions is seeking a Senior Data Engineer to join a major Liquidity Reporting programme, delivering solutions for UK insurers. This hands-on role focuses on building and enhancing a Databricks-based data framework and includes designing scalable PySpark pipelines.

The ideal candidate will have strong experience with Databricks and PySpark, while also providing technical leadership and conducting code reviews to ensure high engineering standards. The position offers a hybrid work arrangement and a day rate of £450-£550.

Qualifications

• Strong experience with Databricks and PySpark.
• Proven ability to build reusable frameworks and libraries.
• Experience leading code quality and working with offshore teams.
• Experience in regulatory reporting or financial services.
• Knowledge of metadata-driven or configurable data models.

Responsibilities

• Design and develop scalable PySpark pipelines covering data ingestion and transformation.
• Provide technical leadership and conduct code reviews.
• Guide offshore teams and ensure delivery quality aligns with regulatory requirements.

Skills

Databricks

PySpark

Building reusable frameworks

Code quality leadership

Regulatory reporting experience

Exposure to Tagetik

Knowledge of metadata-driven models

Tools

Tagetik

Job description

We are seeking a Senior Data Engineer / Developer to join a major Liquidity Reporting programme, delivering regulatory reporting solutions for large UK insurers.

This is a hands-on role focused on building and enhancing a Databricks-based data engineering framework. You will design and develop scalable PySpark pipelines covering data ingestion, validation, transformation, and mapping, while creating reusable libraries and ensuring high engineering standards.

You will also provide technical leadership, conducting code reviews, guiding offshore teams, and ensuring delivery quality aligns with architectural and regulatory requirements.

Key Requirements

• Strong experience with Databricks and PySpark
• Proven ability to build reusable frameworks and libraries
• Experience leading code quality and working with offshore teams
• Experience in regulatory reporting or financial services
• Exposure to Tagetik or similar reporting tools
• Knowledge of metadata-driven or configurable data models

This is an opportunity to play a key role in a large-scale data transformation programme with a strong focus on engineering best practice and regulatory delivery.

Senior Data Engineer | Outside IR35 | £450- £550 per day | Hybrid | London