Data Engineer (Databricks)

X4 Technology; Greater London; On-site; GBP 65,000 - 90,000; Full time

Job summary

X4 Technology is seeking an experienced Data Engineer to design and build a cloud-native Azure Databricks platform supporting Endur, trading, reporting and analytics. You will join a fast-moving transformation programme and own key pipelines from PySpark to Delta Lake.

You will migrate legacy Oracle setups to Databricks, drive CI/CD practices and governance, optimise performance and mentor engineers while staying hands-on in a high-performing team.

Qualifications

• Hands-on experience delivering enterprise-scale Azure Databricks platforms.
• Strong PySpark, Python, SQL, Delta Lake and Structured Streaming expertise.
• Experience migrating legacy Oracle platforms to cloud-native Databricks.
• Background in energy trading, commodities or financial services is a plus.

Responsibilities

• Design, build and optimise scalable Azure Databricks pipelines using PySpark, SQL, Delta Lake and Structured Streaming.
• Lead the migration of legacy Oracle platforms into Azure Databricks, delivering high-performance solutions for trading, reporting and analytics.
• Drive engineering best practice across performance, scalability, CI/CD, testing, governance and cloud cost.
• Work closely with architects, platform teams and stakeholders to shape solutions and mentor engineers.

Skills

PySpark

Python

SQL

Delta Lake

Structured Streaming

Cloud data architecture

Azure Databricks

Tools

Azure Databricks

Job description

Contract: Initial 6-month contract (long-term transformation programme)

X4 Technology are partnered with a leading global energy trading organisation on a major tech transformation - migrating from a legacy ETRM platform onto Endur, and modernising its enterprise data platform with Azure Databricks.

As part of this multi-year programme, Oracle-based data platforms are being rebuilt into a scalable, cloud-native Azure Databricks platform supporting Endur, global reporting, analytics and risk - cutting critical data latency from 10–15 minutes down to near real-time.

They're after an experienced Data Engineer to help design and build that platform hands-on - combining deep engineering with real technical ownership, shaping standards, performance and future architecture along the way.

Responsibilities of the Data Engineer (Azure Databricks)

• Design, build and optimise scalable Azure Databricks pipelines using PySpark, SQL, Delta Lake and Structured Streaming
• Lead the migration of legacy Oracle platforms into Azure Databricks, delivering high-performance solutions for trading, reporting and analytics
• Drive engineering best practice across performance, scalability, CI/CD, testing, governance and cloud cost
• Work closely with architects, platform teams and stakeholders to shape solutions, mentor engineers and influence platform direction

Requirements for the Data Engineer (Azure Databricks)

• Extensive hands-on experience delivering enterprise-scale Azure Databricks platforms - strong PySpark, Python, SQL, Delta Lake and Structured Streaming (Delta Live Tables a bonus)
• Strong grasp of modern cloud data architecture, modelling, performance optimisation, governance and best practice
• Experience on large-scale cloud migration or platform modernisation programmes, with the ability to influence direction and mentor others
• Energy trading, commodities, capital markets or financial services background - a plus, not a must

A brilliant chance to work on one of the biggest Azure Databricks transformations in energy trading, helping build a modern cloud platform underpinning trading, risk, reporting and analytics - with real influence over technical decisions while staying hands-on in a high-performing team.

Interviews moving fast, with a view to onboarding ASAP.