Lead Data Engineer

Stott and May Inc.; Greater London; Remote; GBP 120,000 - 166,000; Part time

Job summary

Stott and May is seeking a Lead Data Engineer to provide technical leadership and hands-on delivery for a Databricks-centric data platform in the insurance/fintech space.

You will own architecture, strategy and roadmap for scalable data pipelines using Python, PySpark and SQL, while guiding a team of Data Engineers and collaborating with Risk, Quantitative Finance, Compliance and Data Science teams. Strong governance and stakeholder communication are essential.

Qualifications

• 7+ years of data engineering experience.
• Strong Databricks experience with large-scale implementations.
• Advanced Python, PySpark and SQL skills.
• Experience leading or mentoring data engineering teams.
• Knowledge of data governance and regulated environments.

Responsibilities

• Own the Databricks architecture, strategy and roadmap.
• Design, build and operationalise scalable ETL/ELT pipelines using Databricks, Python and SQL.
• Develop and optimise Databricks Lakehouse and Delta Lake solutions.
• Implement data governance, lineage, controls and audit frameworks in a regulated environment.
• Lead, mentor and upskill a team of Data Engineers with technical standards and code reviews.
• Provide technical direction across projects and promote engineering best practices.
• Collaborate with Risk, Quantitative Finance, Compliance and Data Science teams to translate requirements into solutions.
• Communicate complex concepts to senior and executive stakeholders.

Skills

Data engineering

Databricks

Python

PySpark

SQL

Databricks Lakehouse

Delta Lake

Cloud platforms

Leadership

Job description

Lead Data Engineer

Location:UK – Fully Remote

Contract:6-month initial contract

Rate:(Inside IR35)

We are looking for an experienced Lead Data Engineer to join a leading insurance client, providing technical leadership across data engineering initiatives and supporting the development of a high-performing engineering team.

The role is both hands-on and leadership-focused, with Databricks at the centre of the technology stack alongside Python, PySpark and SQL. You will take ownership of the Databricks platform and architecture while working closely with senior stakeholders across Risk, Quantitative Finance, Compliance and Data Science.

Key Responsibilities

• Own the Databricks architecture, strategy and roadmap, ensuring performance, availability and cost efficiency.
• Design, build and operationalise scalable ETL/ELT data pipelines using Databricks, Python and SQL.
• Develop and optimise solutions using Databricks Lakehouse and Delta Lake.
• Implement data governance, lineage, controls and audit frameworks within a regulated environment.
• Lead, mentor and upskill a team of Data Engineers, establishing technical standards and conducting code reviews.
• Provide technical direction across data engineering projects and drive engineering best practices.
• Work closely with Risk, Quantitative Finance, Compliance and Data Science teams to translate business requirements into technical solutions.
• Communicate complex technical concepts and architecture clearly to senior and executive stakeholders.

Required Skills & Experience

• 7+ years' experience in Data Engineering.
• 3+ years' hands-on experience delivering large-scale Databricks implementations within financial services.
• Strong experience within Insurance, Banking, Asset Management or FinTech.
• Advanced Python, PySpark and SQL skills.
• Deep knowledge of Databricks Lakehouse architecture and Delta Lake.
• Experience with at least one major cloud platform: AWS, Azure or GCP.
• Proven experience leading or mentoring Data Engineering teams.
• Strong stakeholder management and communication skills.
• Experience working within regulated environments, with an understanding of data governance and compliance requirements.