Senior Data Engineer

CipherTek Recruitment; City Of London; On-site; GBP 110,000 - 140,000; Full time

Job summary

CipherTek Recruitment is hiring a Senior & Lead Data Engineer in London to build a Databricks lakehouse platform for a high-performance Front office Risk trading environment.

This hands-on role focuses on large-scale distributed data systems with deep Spark knowledge, integrating with AI-driven platforms and working closely with front/middle office users to understand market and credit risk and derivatives.

Qualifications

• Hands-on Spark experience in production environments.
• Experience with data modelling and lakehouse architecture.
• Ability to debug and optimise distributed systems for performance.

Responsibilities

• Define technical approach for migrating historic data to Databricks and manage daily/intraday data flow.
• Build pipelines from bronze to silver/gold/platinum.
• Curation of data products in market/credit risk and historic market data domains.
• Build and optimise Spark pipelines on Databricks.
• Develop a lakehouse platform (Medallion architecture).
• Own data modelling, architecture, and pipeline design.
• Work with large-scale data (TB-PB).
• Drive performance, scalability, and reliability in production.

Skills

Spark

Python

Databricks

Data modelling

Lakehouse architecture

Scala (beneficial)

Debugging distributed systems

Tools

Databricks

Job description

12-month rolling (multi-year programme)

1 day/week - St Paul's, London

We're hiring a Senior & Lead Data Engineer to build a Databricks lakehouse platform in a high-performance, business-critical Front office Risk trading environment.

This is a hands-on engineering role focused on building and optimising large-scale distributed data systems.

This is a highly technical team operating at scale, we're looking for engineers with deep data engineering expertise, strong low-level Spark knowledge, and experience building high-performance systems using modern Databricks and AI-driven platforms.

This is working for a Front office Risk Data team, so you MUST have experience working closely with Front/middle office users and have good domain knowledge covering Market and credit Risk and derivatives

What you'll do

• Defining technical approach for migrating historic data from existing SQL server instances to Databricks then keeping that data flow on a daily/intraday basis.
• Building pipelines from bronze to silver/gold/platinum
• Curation of data products in the market risk, credit risk and historic market data domains.
• Build and optimise Spark pipelines on Databricks
• Develop a lakehouse platform (Medallion architecture)
• Own data modelling, architecture, and pipeline design
• Work with large-scale data (TB-PB)
• Drive performance, scalability, and reliability in production

What we're looking for

• Strong experience running Spark workloads in production
• Proven ability to optimise Spark at scale (Tb/PB datasets)
• Solid Python (Scala beneficial, not essential)
• Experience with data modelling and lakehouse architecture
• Ability to debug and improve performance in distributed systems

Important

• Must have recent, hands-on Spark experience
• Databricks strongly preferred (not essential if Spark depth is very strong)
• Experience supporting AI/ML or advanced analytics platforms is a big plus
• Experience in performance-critical environments

Not a fit if

• Primarily BI / reporting focused
• Spark used only at small scale or outside production
• No experience with performance optimisation in distributed systems

Bottom line

We're looking for engineers who can design, build, and optimise Spark-based systems at scale and operate effectively in a performance-critical environment from day one.