Databricks Lead Engineer - South East London

Anson Mccade; Greater London; Hybrid; GBP 130,000 - 150,000; Full time

Job summary

Anson Mccade is seeking a Senior Databricks Architect to lead the delivery of a modern data platform for a major financial services client. You will own Databricks lakehouse solutions from ingestion through to curated serving layers, implementing Medallion Architecture and Unity Catalog.

You will mentor engineers, establish standards and patterns, and drive scalable, production-grade delivery in a hybrid London-based role with on-site three days per week.

Qualifications

• Two years of hands-on Databricks experience.
• Evidence of delivering Databricks solutions into production.
• Active Databricks Data Engineering/ML/Generative AI cert.
• Experience implementing Medallion Architecture across Bronze, Silver, Gold.
• Hands-on Unity Catalog: design, access controls, lineage, sharing.
• Declarative Pipelines with Expectations; failure handling and observability.
• Knowledge of Auto Loader and Lakeflow Connect; ingestion patterns.
• Production PySpark and SQL development.
• Delta Lake knowledge: MERGE, schema evolution, OPTIMIZE, VACUUM.
• Databricks Workflows, orchestration and CI/CD.
• Monitoring, alerting, replay, backfill and operations support.
• Mentor engineers; act as escalation point.
• RBAC, data masking, Photon, cluster optimisation or cost management beneficial.
• Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase advantageous.

Responsibilities

• Lead delivery of a modern enterprise data platform for a leading financial services organisation.
• Own Databricks lakehouse solutions from ingestion to curated serving layers.
• Implement Medallion Architecture, Unity Catalog and Declarative Pipelines.
• Mentor engineers and drive production-quality delivery.
• Shape engineering standards, reusable patterns and delivery practices.

Skills

Databricks

Data engineering

Machine learning

Generative AI

Medallion Architecture

Unity Catalog

Declarative Pipelines

Pyspark

SQL

Delta Lake

RBAC

Photon

Model serving

RAG

Vector Search

Lakebase

CI/CD

Cluster optimisation

Cost management

Mosaic AI

Tools

Auto Loader

Lakeflow Connect

Unity Catalog

Databricks Workflows

Job description

Salary: £130,000 - 149,500 per year

Requirements:

• At least two years of recent, hands-on Databricks experience
• Evidence of delivering Databricks solutions into production environments
• An active Databricks Data Engineering, Machine Learning or Generative AI certification
• Strong experience implementing Medallion Architecture across Bronze, Silver and Gold layers
• Hands-on Unity Catalog experience covering catalog design, access controls, lineage and secure data sharing
• Experience developing Declarative Pipelines with Expectations, including failure handling and observability
• Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns
• Production-level PySpark and SQL development skills
• Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM
• Experience with Databricks Workflows, orchestration and CI/CD
• Understanding of monitoring, alerting, replay, backfill and operational support strategies
• Ability to mentor engineers and act as the technical escalation point
• Experience with RBAC, data masking, Photon, cluster optimisation or cost management would be beneficial
• Exposure to Mosaic AI, Vector Search, model serving, RAG or Lakebase would be advantageous

Responsibilities:

• Lead the delivery of a modern enterprise data platform for a leading financial services organisation
• Take technical ownership of Databricks lakehouse solutions, covering ingestion through to curated serving layers
• Implement Medallion Architecture, Unity Catalog, Declarative Pipelines and scalable Databricks-native ingestion patterns
• Mentor engineers and drive production-quality delivery
• Shape engineering standards, reusable patterns and delivery practices

Technologies:

• AI
• CI/CD
• Databricks
• Support
• Machine Learning
• Model Serving
• PySpark
• RAG
• RBAC
• SQL
• Unity
• GameDev

More:

We are offering up to £575 per day inside IR35 on an initial 6-month contract with potential extension. The role is hybrid with three days per week on-site in London and an ASAP start. We are providing technical leadership of a strategic Databricks implementation, with the opportunity to shape engineering standards, reusable patterns and delivery practices for a leading financial services organisation.

last updated 38 week of 2026