Data Engineer (h/f)

Emagine Consulting; City of Westminster; On-site; GBP 90,000 - 140,000; Full time

Job summary

Emagine Consulting is seeking two Senior Databricks Data Engineers to design, build and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. You will translate architectural requirements into reliable production solutions with clearly defined data contracts and automated tests.

You'll own end-to-end data pipelines, optimize PySpark workloads, and implement governance with Unity Catalog across AWS/Azure/GCP environments.

Qualifications

• 5+ years of hands-on data engineering experience with Databricks.
• Proven production experience with Delta Lake and data lakehouse architectures.
• Expert in Python (PySpark) and SQL.
• Experience with Spec-Driven Development (SDD) and metadata-driven data platforms.

Responsibilities

• Design, build and scale Databricks data pipelines using Medallion Architecture.
• Lead Spec-Driven Development (SDD) by creating machine-readable specifications.
• Develop, monitor, and optimize Airflow-based workflows.
• Create modular dbt models integrated with Databricks SQL/Spark.
• Optimize PySpark workloads and manage cloud compute/storage costs.
• Implement data governance with Unity Catalog and data lineage.

Skills

Databricks

Delta Lake

Apache Airflow

dbt

Python (PySpark)

SQL

Git

CI/CD pipelines

Infrastructure as code

Multi-Cloud (AWS, Azure, GCP)

AWS

Azure

GCP

Tools

Databricks

Delta Lake

Apache Airflow

dbt

Power BI

Terraform

CI/CD tooling

Job description

• Design and develop scalable Databricks data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake.
• Lead the adoption of Spec-Driven Development (SDD) by creating machine-readable specifications and data contracts before implementation.
• Build, monitor, and optimize workflow orchestration pipelines using Apache Airflow.
• Develop, test, and maintain modular data transformation models with dbt, integrated with the Databricks SQL and Spark ecosystem.
• Optimize PySpark workloads, manage cluster configurations, and control compute and storage costs.
• Implement data governance, granular access controls, and data lineage tracking through Unity Catalog.
  5+ years of hands-on data engineering experience, including extensive production experience with Databricks.
• Strong expertise in Apache Airflow for workflow scheduling, dependency management, and error handling.
• Advanced knowledge of dbt modelling, testing practices, and macro development.
• Practical experience with Spec-Driven Development (SDD) and metadata-driven data platform design.
• Expert-level proficiency in Python (PySpark) and SQL.
• Familiarity with Git, CI/CD processes, and infrastructure-as-code practices.
• Multi-Cloud Experience (AWS, Azure, GCP)
• AWS: Configure and manage integrations with Amazon S3, IAM roles, and VPCs to support secure and scalable data platforms.
• Azure: Implement secure data storage using Azure Data Lake Storage Gen2 (ADLS Gen2), integrate with Microsoft Entra ID, and deliver data to analytics platforms such as Power BI.
• GCP: Build and operate data workflows using Google Cloud Storage (GCS) while enabling integration with BigQuery and Vertex AI for downstream analytics and AI use cases.
  emagine is a high-end professional services consultancy and solutions firm specialising in providing business and technology services to the financial services sector, we power progress, solve challenges and deliver real results through tailored high-end consulting services and solutions.
• We have created a culture of openness and integrity by building genuine and strong relationships and partnerships, enabling us to be uncompromising in our dedication in delivering the optimal service for our clients. Our commitment is not just towards our clients but we aim to foster a positive and equitable working environment with our consultants and colleagues which stems from our core values: Confident, Dedicated, Responsible, Genuine.
• 2 Databricks data Engineers: We are seeking 2 Senior Databricks Data Engineers to design, build, and scale a modern Lakehouse platform using Spec-Driven Development (SDD), Apache Airflow, and dbt. In this role, you will translate architectural requirements into reliable production solutions, ensuring data contracts, transformations, and orchestration processes are clearly defined, thoroughly tested, and fully automated.