Lead Databricks Data Engineer — Streaming Platform

Harnham - Data & Analytics Recruitment; Greater London; Hybrid; GBP 111,000 - 129,000; Part time

Job summary

Harnham - Data & Analytics Recruitment is seeking a Lead Databricks Data Engineer for a London-based role with hybrid working. The focus is modernising a data platform for a large financial services organisation, moving from batch processing to streaming-enabled analytics.

You will own the data vision, design the Databricks medallion architecture (Gold layer), and drive delivery with onshore and offshore teams, reporting to senior stakeholders.

Qualifications

• Lead data engineer with experience in financial services or regulated environments.
• Proven capability delivering modern data platforms using Databricks.
• Experience translating business needs into technical data solutions.

Responsibilities

• Define end-to-end data vision and target architecture.
• Lead design and development of a Databricks-based medallion architecture (Gold focus).
• Shift data ingestion from monthly batch to streaming where appropriate.
• Act as senior technical decision-maker on data use cases and trade-offs.
• Coordinate with onshore and offshore teams and accelerate delivery.
• Provide regular updates to senior stakeholders on progress and next steps.

Skills

Lead Data Engineer

Databricks

Data architecture

Stakeholder communication

Team leadership

Tools

Medallion architecture

Job description

Harnham - Data & Analytics Recruitment is seeking a Lead Databricks Data Engineer for a London-based role with hybrid working. The focus is modernising a data platform for a large financial services organisation, moving from batch processing to streaming-enabled analytics.

You will own the data vision, design the Databricks medallion architecture (Gold layer), and drive delivery with onshore and offshore teams, reporting to senior stakeholders.