Senior Data Engineer — Databricks & Data Platform

STACK Infrastructure EMEA; Greater London; On-site; GBP 70,000 - 110,000; Full time

Job summary

STACK Infrastructure EMEA is seeking a Senior Data Engineer to build and manage our data platform on Databricks in the UK. You will develop reliable pipelines from multiple business systems, ensure data quality, governance, and security, and enable robust reporting and analytics for decision-making.

You will design scalable data models, manage Databricks workflows, Unity Catalog and SQL Warehouses, and collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements into

Qualifications

• Significant experience as a Data Engineer with ownership of production data pipelines.
• Strong hands-on experience with Databricks, Delta Lake, Python, SQL and PySpark.
• Experience integrating data from APIs, business applications, databases and files.
• Experience with Unity Catalog, access management, data governance and data quality controls.

Responsibilities

• Design, build and maintain data pipelines into Databricks from APIs, databases, SaaS apps, files and spreadsheets.
• Develop scalable data models and Delta Lake architecture for reporting, analytics and AI.
• Manage Databricks Workflows, Lakehouse components, SQL Warehouses and related jobs.
• Administer Databricks workspaces, Unity Catalog, permissions and service principals.
• Support development, test and production environments including deployment and monitoring.
• Monitor platform performance, usage and costs with security and controls.
• Implement data quality checks for completeness, accuracy and timeliness.
• Ensure dashboards use governed, reliable data and establish data ownership and governance standards.
• Build monitoring to identify pipeline failures and data quality issues; integrate with Power BI and analytics tools.
• Collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements into data solutions.
• Document pipelines, controls and operating procedures; coordinate external partners as needed.

Skills

Databricks

Delta Lake

Python

SQL

PySpark

Unity Catalog

Azure data services

Power BI

Git

CI/CD

Job description

STACK Infrastructure EMEA is seeking a Senior Data Engineer to build and manage our data platform on Databricks in the UK. You will develop reliable pipelines from multiple business systems, ensure data quality, governance, and security, and enable robust reporting and analytics for decision-making.

You will design scalable data models, manage Databricks workflows, Unity Catalog and SQL Warehouses, and collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements into