Principal Data Engineer (Databricks)

X4 Technology; Greater London; Hybrid; GBP 157,000 - 175,000; Full time

Job summary

X4 Technology is delivering a major transformation of its Data Exchange platform for a global energy trading firm in London. The role combines hands-on Azure Databricks engineering with ownership of data architecture, streaming and platform engineering across multiple workstreams.

The Principal Data Engineer will shape the end-to-end data platform, drive standards, and collaborate with architects, engineers and business stakeholders to enable secure, scalable data solutions and explore

Qualifications

• Extensive hands-on Azure Databricks experience with enterprise-scale platforms.
• Proven ownership of data platform architecture including data modelling and governance.
• Experience leading large-scale data transformation programmes across multiple workstreams.
• Strong real-time data and integration experience, including streaming architectures.
• Excellent technical stakeholder management and influencing.

Responsibilities

• Own end-to-end data and platform architecture across multiple workstreams.
• Lead design/delivery of the Databricks data platform - batch and real-time ingestion, streaming, modelling, integration across Azure.
• Set technical direction and engineering standards for performance, security and cost optimisation.
• Be the go-to person across Architects, Engineers, Programme Managers and Trading stakeholders.
• Identify opportunities for AI and automation (MLflow, Azure AI Foundry, Copilot).

Skills

Azure Databricks

PySpark

Python

SQL

Delta Lake

Structured Streaming

Data modelling

Governance

Security

Tools

Kafka

Delta Live Tables

Job description

Programme: Data Exchange Platform

Industry: Commodities / Energy Trading

Location: London (Hybrid)

Contract: Initial 6-month contract (long-term transformation programme)

X4 Technology are partnered with a global energy trading firm on a major transformation of its Data Exchange platform - the technical data backbone supporting physical commodities trading on a global scale.

The platform automates, reconciles and standardises transaction workflows across physical crude oil and refined petroleum products, giving traders a single, trusted source of truth across the entire ecosystem.

Joining as contract Principal Data Engineer, you'll play a pivotal role in the transformation - combining deep Azure Databricks expertise with genuine technical ownership across data architecture, streaming, integration and platform engineering.

Responsibilities for the Principal Data Engineer (Databricks)

• Own end-to-end data and platform architecture, shaping scalable, secure and high-performance solutions across multiple workstreams
• Lead design/delivery of the Databricks data platform - batch and real-time ingestion, streaming, data modelling, integration across Azure
• Set technical direction and engineering standards, driving best practice on performance, scalability, governance, security, testing and cost optimisation
• Be the go-to across Architects, Engineers, Programme Managers, Business Analysts and Trading stakeholders
• Spot and land pportunities for AI and intelligent automation - MLflow, Azure AI Foundry, Agent Bricks and Microsoft Copilot

Requirements for the Principal Data Engineer (Databricks)

• Extensive hands-on Azure Databricks expertise with a strong track record designing and delivering enterprise-scale data platforms (PySpark, Python, SQL, Delta Lake, Structured Streaming, DLT)
• Proven ownership of data platform architecture - data modelling, integration, governance, security, performance and cost optimisation
• Experience leading complex, greenfield or large-scale data transformation programmes across multiple workstreams and teams
• Strong real-time data and integration experience, including streaming architectures and complex data flows (Kafka beneficial)
• Excellent technical stakeholder management and influencing

High-profile seat at the heart of a global trading data ecosystem. You'll combine hands-on data engineering with real technical influence, shaping a highly secure, cloud-native data platform that connects trading desks, physical assets and external partners

Initial 6-month contract with multi-year programme scope, £850-950/day (inside IR35), hybrid in London. Interviews are being scheduled immediately, with a view to onboarding as soon as possible.