Senior Data Engineer: Azure Databricks Platform Lead

Talensa; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Competitive Salary

Annual Bonus and Growth Share Scheme

Pension

Medical and Critical Illness Insurance

Generous Holiday days

Specialist finance business team

Job summary

Talensa, partnered with a specialist B2B Financial Services tech platform, is seeking a Senior / Data Engineer to own an Azure Databricks-based data platform and BI layer. You will build and evolve cloud data pipelines, ensure data quality, and collaborate with platform engineering, data science, and business stakeholders.

The role is hands-on, requires independence, and offers hybrid working: 3 days in the City office and 2 days remote, with opportunities to leverage AI tooling in data

Qualifications

• Commercial experience as a Data Engineer working with Spark in production (ideally on Databricks).
• Experience in finance with strong financial terminology and stakeholder needs.
• Understanding of modern software platform architecture and the data platform role.
• Strong Scala skills (or PySpark with willingness to work in Scala).
• Strong SQL skills for transformations and data modeling.
• Experience building and supporting end-to-end ingestion pipelines.
• Understanding of analytics data modeling (star schema, facts/dimensions).
• Exposure to bronze/silver/gold data architectures and data governance basics.
• Experience in CI/CD and infra-as-code (GitHub Actions, Terraform on Azure).
• Track record of well-tested, maintainable data pipelines.

Responsibilities

• Own and maintain data pipelines ingesting from various sources into Databricks.
• Collaborate with software engineers to evolve the platform and address challenges.
• Design, develop, and optimize Spark/Scala jobs for ingestion and transformation with GitHub Actions/Terraform deployments.
• Implement, run and monitor dbt tests for data quality and dimensional models.
• Extend and maintain Delta Lake fact and dimension tables for analytics use cases.
• Work with analysts, engineers, and business users to ensure trusted datasets.
• Monitor, troubleshoot, and improve pipeline reliability, performance and cost.
• Document pipelines, datasets, and data contracts; be data engineering lead.
• Investigate and resolve data quality incidents and pipeline failures.
• Drive adoption of Databricks features and AI tools into workflows.

Skills

Spark

Scala

SQL

Data pipelines

CI/CD

GitHub Actions

Terraform

Databricks

Delta Lake

Sigma

Tools

Azure Databricks

Confluent Kafka

dbt

Sigma

Git

GitHub Actions

Terraform

SQL Server

Job description

Talensa, partnered with a specialist B2B Financial Services tech platform, is seeking a Senior / Data Engineer to own an Azure Databricks-based data platform and BI layer. You will build and evolve cloud data pipelines, ensure data quality, and collaborate with platform engineering, data science, and business stakeholders.

The role is hands-on, requires independence, and offers hybrid working: 3 days in the City office and 2 days remote, with opportunities to leverage AI tooling in data