Data Engineer - TWE46775

twentyAI; City Of London; On-site; GBP 77,000 - 129,000; Full time

Job summary

twentyAI is hiring a Data Engineer in London to design and build the firm’s first Databricks environment on Azure, implementing a bronze/silver/gold medallion architecture and delivering a trusted golden data source for Finance, Portfolio Operations, and Compliance.

You will collaborate with senior stakeholders to translate business needs into scalable data pipelines, optimize performance, and ensure data quality ahead of the next fund launch.

Qualifications

• Proven delivery of Databricks/Delta Lake builds.
• Strong PySpark and medallion architecture knowledge.
• Azure data platform experience.

Responsibilities

• Design and build the firm's first Databricks instance on Azure.
• Own the technical delivery of the data platform end-to-end.

Skills

Databricks

Delta Lake

Medallion architecture

PySpark

Azure

Stakeholder management

Private Equity experience

Tools

Azure Analysis Services

Job description

Data Engineer – Databricks, Azure, Python, SQL - 12 Month Contract - Outside IR35 - Private Equity - Up to £700 p/d (dependent on experience)

A well-established private equity firm with close to two decades' track record and preparing to launch its largest flagship fund to date is hiring a Data Engineer to build its first Databricks environment from the ground up.

This is a rare opportunity to own the technical delivery of a firm's core data platform end-to-end: partnering directly with stakeholders across Finance, Portfolio Operations, General Partners, and Compliance to unlock the data quality and visibility the business needs ahead of its next fund launch.

The Role

You’ll design and build the firm's first Databricks instance, standing up a medallion architecture (bronze/silver/gold) on Azure and integrating it with the firm's existing portfolio administration and data intelligence platforms. The environment you build will become the firm's golden data source.

What We’re Looking For

• Proven, hands-on delivery of Databricks / Delta Lake builds - ACID transactions, time travel, schema evolution, and performance tuning (OPTIMIZE, Z-ordering, VACUUM)
• Strong command of medallion architecture and PySpark, lazy evaluation, partitioning, broadcast joins, and diagnosing partition skew
• Solid Azure data platform experience; familiarity with Azure Analysis Services or a comparable semantic layer is a plus
• Confidence engaging senior, non-technical stakeholders and translating between technical and business needs
• Experience working within Private Equity is highly desirable - wider Financial Services is accepted

*Suitable candidates will be contacted within 24 hours