Senior Databricks Data Engineer | Lakehouse & Pipelines

STACK Infrastructure; Greater London; On-site; GBP 90,000 - 120,000; Full time

Job summary

STACK Infrastructure in the United Kingdom is seeking a Senior Databricks Data Engineer to build and manage STACK's enterprise data platform. The role emphasizes practical Databricks, PySpark, Python, SQL and Delta Lake to deliver production-grade pipelines and reliable data assets.

You will design scalable lakehouse architectures, govern data with Unity Catalog, and collaborate with Finance, Delivery, Ops, HR and PMO to translate requirements into robust data solutions.

Qualifications

• Production data pipelines built in Databricks with PySpark, Python and SQL.
• Experience with Delta Lake data models and lakehouse architecture.
• Ability to govern data with Unity Catalog and implement data quality checks.

Responsibilities

• Design, develop, deploy and maintain production-grade ETL/ELT pipelines using Databricks, PySpark, Python and SQL.
• Develop and optimize Delta Lake data models for reporting and analytics.
• Build, schedule, monitor and troubleshoot data processing jobs with Databricks tools.
• Apply Unity Catalog governance, including access control and lineage.
• Support Databricks workspaces, compute resources and SQL Warehouses.
• Manage deployments, monitoring, incident resolution and CI/CD practices.
• Optimize performance and control costs while ensuring security and governance.
• Implement data quality checks and monitor data quality and reconciliation with source systems.
• Integrate datasets with Power BI and other analytical tools.
• Collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

CI/CD

Data governance

Data quality

Stakeholder communication

Tools

Databricks Workflows

Delta Live Tables

Lakeflow

Unity Catalog

Power BI

Git

Job description

STACK Infrastructure in the United Kingdom is seeking a Senior Databricks Data Engineer to build and manage STACK's enterprise data platform. The role emphasizes practical Databricks, PySpark, Python, SQL and Delta Lake to deliver production-grade pipelines and reliable data assets.

You will design scalable lakehouse architectures, govern data with Unity Catalog, and collaborate with Finance, Delivery, Ops, HR and PMO to translate requirements into robust data solutions.