Senior Data Engineer: Databricks + PySpark Lead

Opus Recruitment Solutions; Greater London; Hybrid; GBP 83,025 - 101,475; Part time

Job summary

Opus Recruitment Solutions is seeking a Senior Data Engineer to join a major Liquidity Reporting programme, delivering solutions for UK insurers. This hands-on role focuses on building and enhancing a Databricks-based data framework and includes designing scalable PySpark pipelines.

The ideal candidate will have strong experience with Databricks and PySpark, while also providing technical leadership and conducting code reviews to ensure high engineering standards. The position offers a hybrid work arrangement and a day rate of £450-£550.

Qualifications

• Strong experience with Databricks and PySpark.
• Proven ability to build reusable frameworks and libraries.
• Experience leading code quality and working with offshore teams.
• Experience in regulatory reporting or financial services.
• Knowledge of metadata-driven or configurable data models.

Responsibilities

• Design and develop scalable PySpark pipelines covering data ingestion and transformation.
• Provide technical leadership and conduct code reviews.
• Guide offshore teams and ensure delivery quality aligns with regulatory requirements.

Skills

Databricks

PySpark

Building reusable frameworks

Code quality leadership

Regulatory reporting experience

Exposure to Tagetik

Knowledge of metadata-driven models

Tools

Tagetik

Job description

Opus Recruitment Solutions is seeking a Senior Data Engineer to join a major Liquidity Reporting programme, delivering solutions for UK insurers. This hands-on role focuses on building and enhancing a Databricks-based data framework and includes designing scalable PySpark pipelines.

The ideal candidate will have strong experience with Databricks and PySpark, while also providing technical leadership and conducting code reviews to ensure high engineering standards. The position offers a hybrid work arrangement and a day rate of £450-£550.