Senior Databricks Data Engineer

Tenth Revolution Group; United Kingdom; Remote; GBP 90,000 - 130,000; Full time

Job summary

The Senior Databricks Data Engineer will design, build and optimise scalable data pipelines on Databricks, delivering a modern lakehouse to support analytics and reporting for a fast-growing, private equity backed SaaS business.

Working remotely across the UK, you will take ownership from day one, collaborate with analytics, product and business stakeholders, and contribute to CI/CD, governance and data quality initiatives throughout the contract.

Qualifications

• Significant commercial experience as a Data Engineer, with recent hands-on Databricks delivery in production.
• Strong Python and PySpark, with advanced SQL
• Deep understanding of Delta Lake, Spark internals and performance optimisation
• Experience with Unity Catalog and Databricks governance and security features
• Experience building and orchestrating pipelines with Databricks Workflows, Delta Live Tables or tools such as Airflow or Azure Data Factory
• Hands-on experience with a major cloud platform (Azure, AWS or GCP)
• Solid grounding in data modelling (dimensional modelling, data warehousing concepts)
• Experience with Git and CI/CD pipelines (Azure DevOps, GitHub Actions or similar)
• Strong communication skills, with the ability to work autonomously and engage stakeholders in a remote setting

Responsibilities

• Design, build and maintain scalable, reliable data pipelines on Databricks using PySpark and Spark SQL
• Implement and evolve a lakehouse architecture on Delta Lake, following Medallion (bronze, silver, gold) principles
• Ingest data from a range of sources, including SaaS applications, APIs, relational databases and flat files, in batch and near real time
• Build and optimise ETL/ELT workflows using Databricks Workflows, Delta Live Tables and Auto Loader
• Model clean, well-documented datasets that serve BI tools, analysts and downstream applications
• Implement data quality checks, monitoring, alerting and observability to ensure trust in the data
• Tune performance and manage compute costs across clusters, jobs and SQL warehouses
• Apply Unity Catalog for governance, access control, lineage and secure data sharing
• Contribute to CI/CD, version control and infrastructure-as-code practices for data engineering
• Document solutions and hand over knowledge to permanent team members before contract end
• Provide technical guidance and code review, helping to raise standards within the wider team

Skills

Python

PySpark

SQL

Data modelling

Tools

Databricks

Delta Live Tables

Airflow

Azure Data Factory

Delta Lake

Unity Catalog

Job description

Engagement: Contract, initial 3 months, with strong potential to extend

Location: Fully remote, UK-wide

Client: Private equity backed SaaS business

Start: January Start

The Opportunity

Our client is a fast-growing, private equity backed SaaS business investing heavily in its data platform. They are looking for an experienced Senior Databricks Data Engineer to join on an initial 3-month contract, helping to design, build and optimise a modern lakehouse that underpins analytics, reporting and data-driven decision making across the organisation.

You will work as a hands-on senior engineer, taking ownership of delivery from day one, raising engineering standards and working closely with analytics, product and business stakeholders in a collaborative, remote-first environment.

Key Responsibilities

• Design, build and maintain scalable, reliable data pipelines on Databricks using PySpark and Spark SQL
• Implement and evolve a lakehouse architecture on Delta Lake, following Medallion (bronze, silver, gold) principles
• Ingest data from a range of sources, including SaaS applications, APIs, relational databases and flat files, in batch and near real time
• Build and optimise ETL/ELT workflows using Databricks Workflows, Delta Live Tables and Auto Loader
• Model clean, well-documented datasets that serve BI tools, analysts and downstream applications
• Implement data quality checks, monitoring, alerting and observability to ensure trust in the data
• Tune performance and manage compute costs across clusters, jobs and SQL warehouses
• Apply Unity Catalog for governance, access control, lineage and secure data sharing
• Contribute to CI/CD, version control and infrastructure-as-code practices for data engineering
• Document solutions and hand over knowledge to permanent team members before contract end
• Provide technical guidance and code review, helping to raise standards within the wider team

Essential Skills and Experience

• Significant commercial experience as a Data Engineer, with recent hands‑on Databricks delivery in production
• Strong Python and PySpark, with advanced SQL
• Deep understanding of Delta Lake, Spark internals and performance optimisation
• Experience with Unity Catalog and Databricks governance and security features
• Experience building and orchestrating pipelines with Databricks Workflows, Delta Live Tables or tools such as Airflow or Azure Data Factory
• Hands‑on experience with a major cloud platform (Azure, AWS or GCP)
• Solid grounding in data modelling (dimensional modelling, data warehousing concepts)
• Experience with Git and CI/CD pipelines (Azure DevOps, GitHub Actions or similar)
• Strong communication skills, with the ability to work autonomously and engage stakeholders in a remote setting