Senior Databricks Data Engineer

STACK Infrastructure EMEA; Greater London; On-site; GBP 75,000 - 110,000; Full time

Job summary

STACK Infrastructure EMEA is seeking a Senior Databricks Data Engineer to build and manage the enterprise data platform, applying PySpark, Python, SQL and Delta Lake in production.

You will design reliable pipelines across Finance, Delivery, Operations, HR and PMO, implement governance with Unity Catalog, and optimise performance and costs.

Collaborating with cross-functional teams, you will drive data quality, security and scalability, and document pipelines and procedures.

Qualifications

• Hands-on experience building and supporting production data pipelines in Databricks.
• Proficiency with PySpark, Python, SQL and Delta Lake.
• Experience with Unity Catalog governance and data quality practices.
• Experience integrating data from APIs, databases and files.

Responsibilities

• Design, develop and maintain production ETL/ELT pipelines using Databricks, PySpark, Python and SQL.
• Develop and optimize Delta Lake data models and lakehouse architecture for analytics.
• Build, schedule and monitor data processing jobs with Databricks Workflows/Delta Live Tables.
• Apply Unity Catalog governance, including access controls and lineage.
• Support Databricks workspaces, compute resources and SQL Warehouses with IT partnerships.
• Manage deployments, monitoring and incident recovery using Git and CI/CD.
• Optimize pipeline performance and costs while ensuring security and controls.
• Implement data quality checks for completeness, accuracy and timeliness.
• Integrate datasets with Power BI and other analytics tools.
• Collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements.
• Document pipelines, dependencies and operating procedures.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Data governance

CI/CD practices

Tools

Databricks Workflows

Delta Live Tables

Lakeflow

Unity Catalog

Power BI

Job description

Role Summary

We are looking for a Senior Databricks Data Engineer with proven hands-on Databricks experience to build, develop and manage STACK's enterprise data platform. This is a production-focused engineering role requiring practical expertise in Databricks, PySpark, Python, SQL and Delta Lake.

The role will design reliable pipelines integrating Finance, Delivery, Operations, HR and PMO systems, establish governed and high-quality datasets, and support secure, scalable analytics and reporting. Working with technical and business stakeholders, the successful candidate will take ownership of data engineering solutions and help improve the reliability and performance of the platform.

Key Responsibilities:

• Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL, integrating APIs, databases, SaaS applications, files and spreadsheets.
• Develop and optimize scalable Delta Lake data models and lakehouse architecture for reporting, analytics and future AI use cases.
• Build, schedule, monitor and troubleshoot data processing jobs using Databricks Workflows, Lakeflow or Delta Live Tables, as appropriate.
• Apply Unity Catalog governance, including access permissions, data ownership, lineage and common data definitions.
• Support Databricks workspaces, compute resources, SQL Warehouses, service principals and access management in partnership with relevant technology teams.
• Manage development, test and production deployments, monitoring, incident resolution and recovery, using Git and CI/CD practices.
• Optimise pipeline performance, Spark processing, compute usage and platform costs while maintaining appropriate security and operational controls.
• Implement data quality checks for completeness, accuracy, timeliness, consistency, duplicates and reconciliation with source systems.
• Build monitoring and alerting to identify pipeline failures, data quality issues and operational risks.
• Integrate reliable, governed Databricks datasets with Power BI and other approved analytical tools.
• Work with Finance, Delivery, Operations, HR and PMO teams to translate requirements into sustainable data solutions.
• Document pipelines, dependencies, controls and operating procedures, and coordinate specialist external partners when required.

Qualifications:

• Strong hands-on experience building and supporting production data pipelines directly in Databricks; experience with other cloud data platforms alone is not sufficient.
• Practical experience with PySpark, Python, SQL and Delta Lake, including data transformation and performance optimisation.
• Experience orchestrating and troubleshooting Databricks data pipelines using Workflows, Delta Live Tables or Lakeflow.
• Working knowledge of Unity Catalog, data governance, access controls and data quality practices (Essential).
• Experience integrating data from APIs, business applications, databases and files.
• Experience supporting Databricks workspaces, compute resources or SQL Warehouses in a production environment.
• Experience with Azure data services, Power BI, Git and CI/CD practices.
• Understanding of environment management, monitoring, troubleshooting, security and cost control.
• Ability to work with non-technical stakeholders, explain data issues clearly and document solutions effectively.
• Strong ownership, prioritisation and problem-solving skills.

Certifications (Preferred)

• Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator (if available).