Senior Databricks Data Engineer: Lakehouse & SDD Expert

Emagine Consulting; City of Westminster; On-site; GBP 90,000 - 140,000; Full time

Job summary

Emagine Consulting is seeking two Senior Databricks Data Engineers to design, build and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. You will translate architectural requirements into reliable production solutions with clearly defined data contracts and automated tests.

You'll own end-to-end data pipelines, optimize PySpark workloads, and implement governance with Unity Catalog across AWS/Azure/GCP environments.

Qualifications

• 5+ years of hands-on data engineering experience with Databricks.
• Proven production experience with Delta Lake and data lakehouse architectures.
• Expert in Python (PySpark) and SQL.
• Experience with Spec-Driven Development (SDD) and metadata-driven data platforms.

Responsibilities

• Design, build and scale Databricks data pipelines using Medallion Architecture.
• Lead Spec-Driven Development (SDD) by creating machine-readable specifications.
• Develop, monitor, and optimize Airflow-based workflows.
• Create modular dbt models integrated with Databricks SQL/Spark.
• Optimize PySpark workloads and manage cloud compute/storage costs.
• Implement data governance with Unity Catalog and data lineage.

Skills

Databricks

Delta Lake

Apache Airflow

dbt

Python (PySpark)

SQL

Git

CI/CD pipelines

Infrastructure as code

Multi-Cloud (AWS, Azure, GCP)

AWS

Azure

GCP

Tools

Databricks

Delta Lake

Apache Airflow

dbt

Power BI

Terraform

CI/CD tooling

Job description

Emagine Consulting is seeking two Senior Databricks Data Engineers to design, build and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. You will translate architectural requirements into reliable production solutions with clearly defined data contracts and automated tests.

You'll own end-to-end data pipelines, optimize PySpark workloads, and implement governance with Unity Catalog across AWS/Azure/GCP environments.