Senior Data Engineer – Azure Databricks

Talensa; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Competitive Salary

Annual Bonus and Growth Share Scheme

Pension

Medical and Critical Illness Insurance

Generous Holiday days

Specialist finance business team

Job summary

Talensa, partnered with a specialist B2B Financial Services tech platform, is seeking a Senior / Data Engineer to own an Azure Databricks-based data platform and BI layer. You will build and evolve cloud data pipelines, ensure data quality, and collaborate with platform engineering, data science, and business stakeholders.

The role is hands-on, requires independence, and offers hybrid working: 3 days in the City office and 2 days remote, with opportunities to leverage AI tooling in data

Qualifications

• Commercial experience as a Data Engineer working with Spark in production (ideally on Databricks).
• Experience in finance with strong financial terminology and stakeholder needs.
• Understanding of modern software platform architecture and the data platform role.
• Strong Scala skills (or PySpark with willingness to work in Scala).
• Strong SQL skills for transformations and data modeling.
• Experience building and supporting end-to-end ingestion pipelines.
• Understanding of analytics data modeling (star schema, facts/dimensions).
• Exposure to bronze/silver/gold data architectures and data governance basics.
• Experience in CI/CD and infra-as-code (GitHub Actions, Terraform on Azure).
• Track record of well-tested, maintainable data pipelines.

Responsibilities

• Own and maintain data pipelines ingesting from various sources into Databricks.
• Collaborate with software engineers to evolve the platform and address challenges.
• Design, develop, and optimize Spark/Scala jobs for ingestion and transformation with GitHub Actions/Terraform deployments.
• Implement, run and monitor dbt tests for data quality and dimensional models.
• Extend and maintain Delta Lake fact and dimension tables for analytics use cases.
• Work with analysts, engineers, and business users to ensure trusted datasets.
• Monitor, troubleshoot, and improve pipeline reliability, performance and cost.
• Document pipelines, datasets, and data contracts; be data engineering lead.
• Investigate and resolve data quality incidents and pipeline failures.
• Drive adoption of Databricks features and AI tools into workflows.

Skills

Spark

Scala

SQL

Data pipelines

CI/CD

GitHub Actions

Terraform

Databricks

Delta Lake

Sigma

Tools

Azure Databricks

Confluent Kafka

dbt

Sigma

Git

GitHub Actions

Terraform

SQL Server

Job description

Senior / Data Engineer - Financial Services / FinTech Platform - Data Infrastructure

Mid-Senior

London, City

Hybrid working - 3 days a week in City office, 2 WFH

Talensa are partnered with a specialist B2B Financial Services Tech platform firm to hire an experienced Data Engineer (Mid-Senior) to take end-to-end ownership of a modern data platform built on Azure Databricks, Kafka and with BI-layer.

This person would be the primary Data Engineer, maintaining and further developing a well-established cloud and CI/CD setup to implement and evolve reliable, well-tested data pipelines in Spark/Scala and to extend the dimensional data models.

You will be part of the technical team collaborating closely with platform engineering, data science, and business stakeholders to further develop the synergy between the internal software platform and the Databricks data platform.

This role will suits a hands-on Data engineer who is looking to work on interesting Tech, solve data problems, while being a key member of the team, comfortable working independently day to day, following agreed designs, patterns, and standards.

What you’ll do

• Own and maintain data pipelines ingesting from various sources into Databricks
• Collaborate closely with software engineers to evolve our software platform and address technical challenges
• Design, develop, and optimize Spark/Scala jobs for ingestion and transformation, using the existing GitHub Actions and Terraform setup for deployments
• Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models
• Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture
• Work closely with analysts, software engineers, and business users using Sigma to ensure they have trusted, well-documented datasets
• Monitor, troubleshoot, and improve pipeline reliability, performance, and cost within the existing observability framework
• Contribute to documentation of pipelines, datasets, and data contracts, acting as the main point of contact for data engineering topics
• Investigate and resolve data quality incidents and pipeline failures
• Proactively drive adoption of Databricks within the team and experiment with new Databricks features and tools where they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows

Tech stack

• Azure Databricks (Spark, Delta Lake)
• SQL Server, Confluent Kafka
• Scala, SQL, dbt
• Git, GitHub Actions, Terraform
• Sigma (BI)

Must-have experience

• Commercial experience as a Data Engineer working with Spark in production (ideally on Databricks)
• Essential experience working in finance, where you have developed a strong command of financial terminology, concepts, and stakeholder needs to collaborate effectively
• Good understanding of modern software platform architecture and the role of the data platform within the broader platform ecosystem
• Good Scala skills (or strong PySpark plus a genuine willingness to work primarily in Scala)
• Strong SQL skills for transformations, data modelling, and debugging data issues
• Experience building and supporting ingestion pipelines end to end
• Understanding of data modelling for analytics (star schema, facts/dimensions) and how schema changes affect downstream users
• Exposure to layered data architectures (e.g. bronze/silver/gold) and basic data governance practices (permissions, documentation, data ownership
• Experience working in a CI/CD environment and with infra-as-code tools (e.g. GitHub Actions and Terraform on Azure)
• Track record of writing well-tested, maintainable data pipelines and jobs (unit/integration tests and data quality checks)
• Ability to work as a self-starter, taking ownership of the data platform, making pragmatic decisions, and driving work to completion with limited day-to-day supervision

Nice to have

• Experience with Azure data services beyond Databricks (e.g. Key Vault, Storage).
• Experience with modern BI tools (e.g. Sigma, Looker, Power BI, Tableau).
• Experience working with AI coding agents (like Claude Code, Cursor, Copilot, etc) and writing developed thoughtful prompts
• Interest in analytics and data science, and working with others to turn data into insight

What’s on offer:

• Competitive Salary
• Annual Bonus and participation in Growth Share Scheme
• Pension
• Medical and Critical illness Insurance
• Generous Holiday days
• Opportunity to work in a specialist finance business with a great team

To note on scope

This is the only senior / experienced data engineer role in the Engineering Tech team with no immediate plans to hire underneath. Would suit someone who wants to work on interesting Tech & Data problems, platform innovations and support driving wider business growth.