Data Engineer

Cititec; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Job summary

Cititec are partnering with a global commodities trading organisation in London to build a modern Databricks-based Enterprise Data Platform. This hands-on role focuses on real-time data pipelines and streaming integration, with emphasis on platform rollout and governance.

You will join an international delivery function and collaborate with risk, cloud engineering and data architecture teams to evolve the lakehouse, migrate legacy datasets, and deliver business-ready data products.

Qualifications

• Hands-on experience with Databricks workloads, notebooks, and workflows.
• End-to-end real-time streaming delivery across pipelines.
• Experience migrating or integrating legacy data sources.
• Comfortable working in a distributed international team.
• Familiarity with data governance concepts is a plus.

Responsibilities

• Support the build-out and rollout of a new Databricks platform, working alongside an existing analytics environment during the transition.
• Build real-time and streaming data pipelines (e.g. Kafka, Spark Structured Streaming, Delta Live Tables) to support business-critical data flows.
• Contribute to the delivery of a substantial pipeline of data products for business consumption, as part of an early wave of platform rollout.
• Integrate data from across the business, including eventual connection to market risk data feeds.
• Migrate legacy platforms and datasets (e.g. Hadoop, SAP BW, on-prem warehouses) into a modern lakehouse environment.
• Help evolve the platform’s approach to data governance, cataloguing and cross-platform access as the business scales.
• Collaborate with a wider Enterprise Data Platform team, including an international delivery function, and stakeholders across risk, cloud engineering and semantic data teams.
• Work effectively in a fast-moving, still-evolving team structure where priorities and ownership are still being defined.

Skills

Hands-on Databricks experience

Real-time streaming data experience

Tools

Databricks

Kafka

Spark Structured Streaming

Delta Live Tables

Job description

Data Engineer – Databricks & Real-Time Data London (Onsite/Hybrid)

Cititec are working with a global commodities trading organisation that is expanding its Enterprise Data Platform capability as part of a major shift in its data strategy. They are onboarding Databricks as a new central analytics platform to sit alongside their existing data estate, with a view to it eventually sourcing data across the wider business, including future integration with market risk data feeds. This is a hands‑on role for someone who can build real‑time data pipelines and help stand up a platform that is still taking shape, working closely with cloud engineering, risk and data architecture teams.

You’ll be responsible for helping build and roll out the Databricks platform, with a strong focus on real‑time and streaming data integration across the business.

Typical responsibilities include:

• Supporting the build‑out and rollout of a new Databricks platform, working alongside an existing analytics environment during the transition

• Building real‑time and streaming data pipelines (e.g. Kafka, Spark Structured Streaming, Delta Live Tables) to support business‑critical data flows

• Contributing to the delivery of a substantial pipeline of data products for business consumption, as part of an early wave of platform rollout

• Integrating data from across the business, including eventual connection to market risk data feeds

• Migrating legacy platforms and datasets (e.g. Hadoop, SAP BW, on‑prem warehouses) into a modern lakehouse environment

• Helping evolve the platform’s approach to data governance, cataloguing and cross‑platform access as the business scales

• Collaborating with a wider Enterprise Data Platform team, including an international delivery function, and stakeholders across risk, cloud engineering and semantic data teams

• Working effectively in a fast‑moving, still‑evolving team structure where priorities and ownership are still being defined

What We’re Looking For

• Demonstrable, hands‑on Databricks experience is essential: production workloads, notebooks, workflows and cluster management, not a single proof‑of‑concept

• Real‑time / streaming data experience is essential: genuine end‑to‑end streaming pipeline delivery, not solely scheduled batch ETL/ELT

• Background in enterprise data platforms, data engineering or similar, ideally including legacy platform migration into a modern lakehouse

• Comfortable working in a fast‑moving, still‑evolving team structure with a distributed, international delivery team

• Data governance and cataloguing exposure is a plus, such as Unity Catalog, data lineage or data quality frameworks

• Experience in commodities trading, energy trading or financial services is strongly preferred but not required