Lead Data Engineer: Databricks Lakehouse Architect & Mentor

Stott and May; United Kingdom; On-site; GBP 90,000 - 150,000; Full time

Job summary

Stott and May is partnering with a leading insurance client to hire an experienced Lead Data Engineer. You will own the Databricks platform, set the architecture, and guide a high-performing data team.

The role combines hands-on engineering with strategic leadership across Risk, Quantitative Finance, Compliance and Data Science, using Python, PySpark and SQL to deliver scalable data solutions in a regulated environment.

Qualifications

• 7+ years of Data Engineering experience.
• 3+ years hands-on Databricks deployments in financial services.
• Deep experience in Insurance/Banking/FinTech.

Responsibilities

• Own Databricks architecture, strategy and roadmap.
• Design, build and operate scalable ETL/ELT pipelines using Databricks, Python and SQL.
• Develop and optimise Databricks Lakehouse/Delta Lake solutions.
• Implement data governance, lineage, controls and audit frameworks in a regulated environment.

Skills

Databricks

Python

PySpark

SQL

Data engineering

Leadership

Stakeholder mgmt

Tools

Delta Lake

Databricks Lakehouse

Job description

Stott and May is partnering with a leading insurance client to hire an experienced Lead Data Engineer. You will own the Databricks platform, set the architecture, and guide a high-performing data team.

The role combines hands-on engineering with strategic leadership across Risk, Quantitative Finance, Compliance and Data Science, using Python, PySpark and SQL to deliver scalable data solutions in a regulated environment.