Data Engineer for Azure Databricks Platform Transformation

X4 Technology; Greater London; On-site; GBP 65,000 - 90,000; Full time

Job summary

X4 Technology is seeking an experienced Data Engineer to design and build a cloud-native Azure Databricks platform supporting Endur, trading, reporting and analytics. You will join a fast-moving transformation programme and own key pipelines from PySpark to Delta Lake.

You will migrate legacy Oracle setups to Databricks, drive CI/CD practices and governance, optimise performance and mentor engineers while staying hands-on in a high-performing team.

Qualifications

• Hands-on experience delivering enterprise-scale Azure Databricks platforms.
• Strong PySpark, Python, SQL, Delta Lake and Structured Streaming expertise.
• Experience migrating legacy Oracle platforms to cloud-native Databricks.
• Background in energy trading, commodities or financial services is a plus.

Responsibilities

• Design, build and optimise scalable Azure Databricks pipelines using PySpark, SQL, Delta Lake and Structured Streaming.
• Lead the migration of legacy Oracle platforms into Azure Databricks, delivering high-performance solutions for trading, reporting and analytics.
• Drive engineering best practice across performance, scalability, CI/CD, testing, governance and cloud cost.
• Work closely with architects, platform teams and stakeholders to shape solutions and mentor engineers.

Skills

PySpark

Python

SQL

Delta Lake

Structured Streaming

Cloud data architecture

Azure Databricks

Tools

Azure Databricks

Job description

X4 Technology is seeking an experienced Data Engineer to design and build a cloud-native Azure Databricks platform supporting Endur, trading, reporting and analytics. You will join a fast-moving transformation programme and own key pipelines from PySpark to Delta Lake.

You will migrate legacy Oracle setups to Databricks, drive CI/CD practices and governance, optimise performance and mentor engineers while staying hands-on in a high-performing team.