Senior Data Engineer (Databricks, Scala, Spark) - Contract

La Fosse; London; Hybrid; GBP 60,000 - 100,000; Full time

Job summary

An established industry player is seeking a Senior Data Engineer to enhance their data lakehouse capabilities. This exciting role involves designing high-performance data pipelines and collaborating with business users in the Capital Markets sector. You will leverage your expertise in Databricks, Scala, and Python to optimize data integration and maintain data quality. Join a dynamic team where your contributions will directly impact financial services, and enjoy working in a collaborative environment that values innovation and excellence.

Qualifications

• Expertise in Databricks and strong programming skills in Scala and Python.
• Deep understanding of ETL design and data modelling.

Responsibilities

• Design and maintain scalable data pipelines for a financial services client.
• Collaborate with Traders to deliver data solutions aligned with business needs.

Skills

Databricks

Scala

Python

Spark architecture

ETL design

Data modelling

Agile delivery

Trading knowledge

Job description

Location: Central London (2–3 days/week onsite)
Type: Contract – 6 months +

Outside IR35

We’re looking for a Senior Data Engineer with a strong track record in building and maintaining data lakehouses, ideally within banking or capital markets domains to join our global financial services client on a contract basis.

This role supports business users in the Capital Markets space, so knowledge of derivatives, risk, PnL, trade lifecycle, market data, and scenarios is highly desirable.

Key Responsibilities

• Own and drive components of the data platform through deep domain understanding.
• Design, develop, and maintain scalable, high-performance data pipelines.
• Ensure consistent and accurate integration of internal and external data sources.
• Follow best practices in ETL, data modelling (medallion architecture), and pipeline development.
• Collaborate with Traders and Quants to deliver data solutions aligned with business needs.
• Maintain data quality, troubleshoot issues, and optimize platform performance.
• Produce clear documentation for pipeline design and processes.

Required Skills & Experience

• Expertise around Databricks, including unity catalogue (preferably on Azure)
• Strong programming in Scala and Python
• Deep understanding of Spark architecture and performance tuning
• Proven experience with ETL design, data modelling, and database design
• Strong grasp on the full SDLC and Agile delivery
• Solid knowledge of trading, derivatives, and risk systems (nice to have)