Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Venn Group; City Of London; Hybrid; GBP 110,000 - 140,000; Full time

Job summary

Venn Group in London is seeking a Senior Data Engineer to join a global banking organisation, focusing on Counterparty Credit Risk data in a hybrid environment.

You will design and optimise scalable data pipelines using Python, PySpark, Databricks and Lakehouse architectures, handling CCR and Credit Risk data in production.

Qualifications

• 10+ years' experience across Data Engineering or related data disciplines.
• Hands-on Python development with Spark experience.
• Extensive Databricks, notebooks, jobs/workflows and Delta Lake expertise.
• Strong SQL and relational database experience; data modelling and quality controls.

Responsibilities

• Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks.
• Build and optimise ETL/ELT workflows with Spark and Delta Lake.
• Engineer data supporting Counterparty Credit Risk, Credit Risk and related risk functions.
• Work with datasets on counterparties, exposures, limits and risk analytics.
• Design robust Lakehouse architectures and manage Databricks notebooks and jobs.
• Enhance data processing performance, logging, monitoring and governance.
• Integrate data platforms with Power BI dashboards and production systems.
• Collaborate with Risk, Technology and Analytics stakeholders; support CI/CD.

Skills

Python

PySpark

Databricks

SQL

ETL/ELT pipelines

Lakehouse

Power BI

Airflow

Delta Lake

Cloud platforms

Data modelling

CI/CD

Git

Kubernetes

Tools

Databricks

Airflow

CI/CD tooling

Docker

Kubernetes

Job description

Senior Data Engineer - Python, Databricks & Counterparty Credit Risk

Location: London - Hybrid

We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data.

This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks, alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.

Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data, including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.

Key Responsibilities

• Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks
• Build and optimise ETL/ELT workflows using Spark and Delta Lake
• Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions
• Work with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analytics
• Design robust data models and architectures across Lakehouse / Medallion environments
• Develop and manage Databricks notebooks, jobs and workflows
• Optimise large-scale distributed data processing and troubleshoot Spark performance
• Implement strong data quality, reconciliation, governance and monitoring controls
• Build reliable logging and alerting across production data pipelines
• Integrate data platforms with Power BI dashboards and wider applications
• Support CI/CD and engineering best practices across development, testing and production environments
• Work closely with Risk, Technology, Analytics and other senior stakeholders

Required Experience

• 10+ years' experience across Data Engineering or closely related data disciplines
• Strong hands-on Python development experience
• Strong PySpark / Apache Spark engineering experience
• Extensive hands-on experience with Databricks, including notebooks, jobs/workflows and Delta Lake
• Advanced SQL and relational database experience
• Strong track record designing and building scalable ETL/ELT data pipelines
• Experience with Delta Lake, Lakehouse and/or Medallion architectures
• Experience handling banking risk data, ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains
• Strong understanding of data modelling, data quality and production data engineering
• Cloud experience across Azure, AWS and/or GCP
• Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift
• Experience integrating data platforms with Power BI
• Experience with orchestration such as Databricks Workflows or Airflow

Highly Desirable

• Direct experience engineering data for Counterparty Credit Risk, including exposure/counterparty datasets
• Knowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits
• Previous experience within an investment bank, global bank or capital-markets environment
• Databricks Data Engineer certification
• Kafka / Structured Streaming experience
• Delta Lake CDC, Time Travel and optimisation
• CI/CD, Git and DevOps experience
• Docker / Kubernetes experience
• Experience exposing data through APIs or supporting React-based front-end applications
• Experience supporting analytics or ML workloads within Databricks

We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.