Contract Senior Data Engineer (Databricks)

X4 Technology; Greater London; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

X4 Technology is seeking a Senior Data Engineer to design and build an Azure Databricks-based data platform supporting trading, reporting and analytics for a major energy trading transformation in London. The role combines hands-on engineering with technical ownership and collaboration with architects and platform teams.

You will migrate legacy Oracle data platforms to Azure Databricks, implement scalable pipelines with PySpark, Python, SQL and Delta Lake, drive CI/CD and governance, and mentor

Qualifications

• Extensive hands-on experience delivering enterprise-scale Azure Databricks data platforms.
• Strong expertise in PySpark, Python, SQL, Delta Lake and Structured Streaming.
• Experience migrating legacy data platforms to cloud-based modern analytics.
• Experience in energy trading, commodities or financial services is advantageous but not required.

Responsibilities

• Design, build and optimise scalable Azure Databricks data pipelines.
• Lead migration of legacy Oracle data platforms into Azure Databricks.
• Drive best practices for performance, scalability, CI/CD, testing and cost control.
• Collaborate with architects, platform teams and stakeholders, mentoring engineers.

Skills

PySpark

Python

SQL

Delta Lake

Structured Streaming

Azure Databricks

CI/CD

Mentoring

Tools

Delta Live Tables

Azure

Git

Job description

Industry: Commodities / Energy Trading

Location: London (Hybrid)

Contract: Initial 6-month contract (long-term transformation programme)

X4 Technology are partnered with a leading global energy trading organisation delivering a major technology transformation centred on the migration from a legacy ETRM platform to Endur and the modernisation of its enterprise data platform using Azure Databricks.

As part of this multi-year programme, the organisation is migrating Oracle-based data platforms into Azure Databricks to build a scalable, cloud-native data platform supporting Endur, global reporting, analytics and risk management, while reducing critical data latency from 10-15 minutes to near real-time.

They are looking for an experienced Senior Data Engineer to play a key role in designing and building the Azure Databricks platform. This role combines deep hands-on engineering with technical ownership, helping define engineering standards, optimise platform performance and shape the future architecture of a business-critical data platform.

Responsibilities of the Senior Data Engineer (Azure Databricks)

• Design, build and optimise scalable Azure Databricks data pipelines and platforms using PySpark, SQL, Delta Lake and Structured Streaming.
• Lead the migration of legacy Oracle data platforms into Azure Databricks, delivering high-performance data solutions for trading, reporting and analytics.
• Drive engineering best practices across performance, scalability, CI/CD, testing, governance and cloud cost optimisation.
• Collaborate with architects, platform teams and business stakeholders to shape technical solutions, mentor engineers and influence platform direction.

Requirements for the Senior Data Engineer (Azure Databricks)

• Extensive hands-on experience delivering enterprise-scale Azure Databricks data platforms, with strong expertise in PySpark, Python, SQL, Delta Lake and Structured Streaming (Delta Live Tables desirable).
• Strong understanding of modern cloud data architecture, data modelling, performance optimisation, governance and data engineering best practices.
• Experience working on large-scale cloud migration or platform modernisation programmes, with the ability to influence technical direction and mentor engineering teams.
• Experience within energy trading, commodities, capital markets or financial services is advantageous but not essential.

This is an opportunity to work on one of the largest Azure Databricks transformations in the energy trading sector, helping build a modern cloud data platform that underpins trading, risk, reporting and analytics. You'll have significant influence over technical decisions while remaining deeply involved in hands-on engineering within a high-performing team.

Interviews are being scheduled immediately, with a view to onboarding as soon as possible.