Databricks Lead Engineer: Medallion Data Platform

Anson Mccade; Greater London; Hybrid; GBP 130,000 - 150,000; Full time

Job summary

Anson Mccade is seeking a Senior Databricks Architect to lead the delivery of a modern data platform for a major financial services client. You will own Databricks lakehouse solutions from ingestion through to curated serving layers, implementing Medallion Architecture and Unity Catalog.

You will mentor engineers, establish standards and patterns, and drive scalable, production-grade delivery in a hybrid London-based role with on-site three days per week.

Qualifications

• Two years of hands-on Databricks experience.
• Evidence of delivering Databricks solutions into production.
• Active Databricks Data Engineering/ML/Generative AI cert.
• Experience implementing Medallion Architecture across Bronze, Silver, Gold.
• Hands-on Unity Catalog: design, access controls, lineage, sharing.
• Declarative Pipelines with Expectations; failure handling and observability.
• Knowledge of Auto Loader and Lakeflow Connect; ingestion patterns.
• Production PySpark and SQL development.
• Delta Lake knowledge: MERGE, schema evolution, OPTIMIZE, VACUUM.
• Databricks Workflows, orchestration and CI/CD.
• Monitoring, alerting, replay, backfill and operations support.
• Mentor engineers; act as escalation point.
• RBAC, data masking, Photon, cluster optimisation or cost management beneficial.
• Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase advantageous.

Responsibilities

• Lead delivery of a modern enterprise data platform for a leading financial services organisation.
• Own Databricks lakehouse solutions from ingestion to curated serving layers.
• Implement Medallion Architecture, Unity Catalog and Declarative Pipelines.
• Mentor engineers and drive production-quality delivery.
• Shape engineering standards, reusable patterns and delivery practices.

Skills

Databricks

Data engineering

Machine learning

Generative AI

Medallion Architecture

Unity Catalog

Declarative Pipelines

Pyspark

SQL

Delta Lake

RBAC

Photon

Model serving

RAG

Vector Search

Lakebase

CI/CD

Cluster optimisation

Cost management

Mosaic AI

Tools

Auto Loader

Lakeflow Connect

Unity Catalog

Databricks Workflows

Job description

Anson Mccade is seeking a Senior Databricks Architect to lead the delivery of a modern data platform for a major financial services client. You will own Databricks lakehouse solutions from ingestion through to curated serving layers, implementing Medallion Architecture and Unity Catalog.

You will mentor engineers, establish standards and patterns, and drive scalable, production-grade delivery in a hybrid London-based role with on-site three days per week.