Head of Data Engineering

JR United Kingdom; City Of London; On-site; GBP 80,000 - 120,000; Full time

Job summary

A global Private Equity firm is seeking a Head of Data Engineering in London to lead a critical data transformation program. The role focuses on implementing a data-first approach using Databricks on Azure, managing a team, and ensuring data quality and governance while supporting business growth through actionable insights.

Qualifications

• Technical background in data engineering with experience in Python, Spark, Databricks, and Azure cloud.
• Experience managing the end-to-end build of an Azure Databricks platform.
• Experience in Financial Services or similarly regulated industries is critical.

Responsibilities

• Lead a data transformation and modernization program.
• Oversee the progression of Databricks setup from proof-of-concept to production.
• Implement a semantic layer for data management and develop data pipelines.

Skills

Data Quality

Data Governance

Leadership

Management

Communication

Tools

Databricks

Azure

Python

Spark

Job description

Here is a revised version of the job description with improved formatting and clarity:

Head of Data Engineering, London (City of London)

Client: Cornwallis Elt

Location: London (City of London), United Kingdom

Job Category: Other

EU work permit required: Yes

Job Views: 10

Posted: 25.05.2025

Expiry Date: 09.07.2025

Job Description:

Head of Data Engineering – Technologies: Databricks, Azure, Python, ETL, Data Pipelines; Focus: Data Strategy, Data Governance, Leadership, Management, Private Markets

A global Private Equity firm is seeking an experienced Head of Data Engineering to lead a critical data transformation and modernization program, moving towards a data-first approach. The role involves replacing a legacy data warehouse with a platform-as-a-service model using Databricks on Azure, with line management responsibilities for the Data Engineering team.

You will oversee the progression of their Databricks setup from proof-of-concept to production, ensuring high levels of optimization and scalability for global operations. Responsibilities include implementing a semantic layer for data management, developing data pipelines, and enabling real-time reporting and visualization.

Post-platform deployment, the focus will shift to applying Data Science techniques to build a data function that provides deep, actionable insights in partnership with the Investment Team, supporting business growth.

The ideal candidate will demonstrate:

• A technical background in data engineering with experience in Python, Spark, Databricks, and Azure cloud.
• Experience managing the end-to-end build of an Azure Databricks platform.
• A passion for data quality, governance, and the positive impact of good data.
• Experience leading high-performing technology teams and establishing best practices, while engaging technically.
• Ability to confidently communicate with stakeholders at all levels.
• Experience in Financial Services (Private Markets, Investment Banking, Investment Management) or similarly regulated industries such as Gambling/Online Gaming is critical.

If you are passionate about data and leading teams to develop modern data platforms, this is an exciting opportunity to join a growing business transitioning to a data-driven approach.