Senior Databricks Data Engineer — Enterprise Platform Lead

STACK Infrastructure EMEA; Greater London; On-site; GBP 75,000 - 110,000; Full time

Job summary

STACK Infrastructure EMEA is seeking a Senior Databricks Data Engineer to build and manage the enterprise data platform, applying PySpark, Python, SQL and Delta Lake in production.

You will design reliable pipelines across Finance, Delivery, Operations, HR and PMO, implement governance with Unity Catalog, and optimise performance and costs.

Collaborating with cross-functional teams, you will drive data quality, security and scalability, and document pipelines and procedures.

Qualifications

• Hands-on experience building and supporting production data pipelines in Databricks.
• Proficiency with PySpark, Python, SQL and Delta Lake.
• Experience with Unity Catalog governance and data quality practices.
• Experience integrating data from APIs, databases and files.

Responsibilities

• Design, develop and maintain production ETL/ELT pipelines using Databricks, PySpark, Python and SQL.
• Develop and optimize Delta Lake data models and lakehouse architecture for analytics.
• Build, schedule and monitor data processing jobs with Databricks Workflows/Delta Live Tables.
• Apply Unity Catalog governance, including access controls and lineage.
• Support Databricks workspaces, compute resources and SQL Warehouses with IT partnerships.
• Manage deployments, monitoring and incident recovery using Git and CI/CD.
• Optimize pipeline performance and costs while ensuring security and controls.
• Implement data quality checks for completeness, accuracy and timeliness.
• Integrate datasets with Power BI and other analytics tools.
• Collaborate with Finance, Delivery, Operations, HR and PMO to translate requirements.
• Document pipelines, dependencies and operating procedures.

Skills

Databricks

PySpark

Python

SQL

Delta Lake

Data governance

CI/CD practices

Tools

Databricks Workflows

Delta Live Tables

Lakeflow

Unity Catalog

Power BI

Job description

STACK Infrastructure EMEA is seeking a Senior Databricks Data Engineer to build and manage the enterprise data platform, applying PySpark, Python, SQL and Delta Lake in production.

You will design reliable pipelines across Finance, Delivery, Operations, HR and PMO, implement governance with Unity Catalog, and optimise performance and costs.

Collaborating with cross-functional teams, you will drive data quality, security and scalability, and document pipelines and procedures.