Lead Data Engineer — Cloud Data Platform (Databricks/Azure)

Pioneer Search; City Of London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Competitive salary

Bonus

Comprehensive benefits package

Hybrid work from London office

Job summary

Pioneer Search is seeking a Lead Data Engineer in London to design and evolve a modern cloud-based data platform built on Databricks and Azure. You will lead scalable data pipelines, mentor a team of Data Engineers and collaborate with product owners to drive data-enabled decision-making.

The role combines hands-on delivery with strategic engineering leadership, leveraging Python, PySpark and Azure to deliver high-quality data solutions across underwriting and analytics.

Qualifications

• Experience building data engineering solutions with Python, PySpark and Databricks.
• Experience designing complex SQL queries for large datasets.
• Strong Azure stack experience including Azure Functions.
• Experience leading Data Engineers in Agile environments.
• Understanding of distributed data processing and CI/CD.
• Experience building scalable, high-performing data platforms.
• Experience in Financial Services or Insurance is advantageous.

Responsibilities

• Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.
• Design, develop and optimise scalable data pipelines using Python, PySpark and Databricks.
• Build and maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions and related services.
• Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
• Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
• Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
• Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.

Skills

Python

PySpark

Databricks

SQL

Azure

CI/CD

Mentoring

Agile

Data pipelines

Cloud data platform

Tools

Azure Functions

CI/CD tooling

Job description

Pioneer Search is seeking a Lead Data Engineer in London to design and evolve a modern cloud-based data platform built on Databricks and Azure. You will lead scalable data pipelines, mentor a team of Data Engineers and collaborate with product owners to drive data-enabled decision-making.

The role combines hands-on delivery with strategic engineering leadership, leveraging Python, PySpark and Azure to deliver high-quality data solutions across underwriting and analytics.