Databricks RSA - DPP Level - SC Cleared

Consult; Greater London; Hybrid; GBP 120,000 - 166,000; Full time

Benefits offered by this job

Job summary

Consult is seeking a Databricks Resident Solutions Architect (SC Cleared) to work on a UK public sector engagement in London. The role is contract-based with an initial 6 months and strong extension potential, onsite as required with hybrid attendance.

You will design, govern and deliver a robust Databricks Lakehouse, implement Unity Catalog governance, and drive production-grade pipelines while ensuring UK-sovereign data handling and cost visibility. Start ASAP.

Qualifications

• Active UK SC clearance from day one.
• Current DPP vetting as a Databricks Resident Solutions Architect or equivalent status.
• 4+ years hands-on Databricks delivery with at least two end-to-end production implementations as lead/architect.
• Deep Unity Catalog experience with RBAC on sensitive data.
• Strong PySpark, Spark SQL and Delta Lake experience, including performance tuning at scale.
• Production pipelines with Lakeflow, DLT, Workflows and DABs.

Responsibilities

• Lead the architecture and build of a secure Databricks Lakehouse on Azure or AWS, using medallion design (bronze, silver, gold).
• Design and implement Unity Catalog governance: catalogues, schemas, RBAC, row/column security, lineage and audit for government data.
• Build production-grade ingestion and pipelines with Lakeflow Connect, Lakeflow Declarative Pipelines, (DLT) and Workflows; move beyond notebooks to deployable code.
• Establish CI/CD and platform engineering standards using Databricks Asset Bundles (DABs), Terraform and Azure DevOps or GitHub Actions.
• Lead migrations from legacy platforms with clear cut-over and reconciliation plans.
• Monitor and optimise DBU consumption and cluster policies for cost visibility.
• Enable MLOps and GenAI use cases including MLflow, Model Serving and Genie.
• Act as trusted advisor to client stakeholders with workshops and architecture reviews.
• Coach and upskill client and partner engineers towards self-sufficiency.
• Support pre-sales and RFP responses with solution design and estimates.

Job description

Role overview

A Databricks Partner Program (DPP) vetted Resident Solutions Architect with active SC clearance, embedded with a UK public sector client to design, govern and deliver a production Databricks Lakehouse.

Job title

• Databricks Resident Solutions Architect (DPP), SC Cleared
• Engagement: Contract, 6 months initial with strong extension potential
• Clearance: Active SC required from day one
• Location: London
• Hybrid vs remote: client site attendance as required
• Rate: DOE - Outside IR35
• Start: ASAP

About the engagement:

Our client is a UK Databricks partner consultancy delivering data modernisation for central government.

The programme moves sensitive, regulated datasets off legacy warehouses and siloed cloud storage onto a governed Lakehouse. Data must stay within UK sovereign boundaries, access must be tightly controlled, and cost efficiency matters under ongoing public spending pressure. You will be the senior Databricks authority on site, trusted by both the client and Databricks.

Key responsibilities:

• Lead the architecture and build of a secure Databricks Lakehouse on Azure or AWS, using medallion (bronze, silver, gold) design
• Design and implement Unity Catalog governance: catalogues, schemas, role-based access control, row and column level security, lineage and audit for sensitive government data
• Build production-grade ingestion and pipelines with Lakeflow Connect, Lakeflow Declarative Pipelines (DLT) and Workflows, moving beyond notebooks to tested, deployable code
• Establish CI/CD and platform engineering standards using Databricks Asset Bundles (DABs), Terraform and Azure DevOps or GitHub Actions
• Lead migrations from legacy platforms (SQL Server, Oracle, SAS, Hadoop, Synapse) with clear cut-over and reconciliation plans
• Monitor and optimise DBU consumption and cluster policies, giving the client clear cost visibility
• Enable MLOps and GenAI use cases where in scope, including MLflow, Model Serving and Genie
• Act as trusted advisor to client stakeholders, running workshops, design authorities and architecture reviews
• Coach and upskill client and partner engineers, leaving the client self-sufficient
• Support pre-sales and RFP responses with solution design and estimates

Essential skills and experience:

• Active UK SC clearance (please state the sponsoring organisation and expiry date)
• Current DPP vetting as a Databricks Resident Solutions Architect, or equivalent Databricks-approved delivery status
• 4+ years hands-on Databricks delivery, with at least two end-to-end production implementations as lead or architect
• Deep Unity Catalog experience: you have configured it, not just used it, including RBAC on sensitive data
• Strong PySpark, Spark SQL and Delta Lake, including performance tuning at scale
• Production pipeline experience with Lakeflow / DLT, Workflows and DABs

Desirable:

• Databricks Certified Data Engineer Professional, Solutions Architect Champion, or Databricks Champion status
• Databricks Summit, user group or community speaking
• MLflow, Model Serving, Mosaic AI or Genie delivery experience
• Lakebase, Delta Sharing or Lakehouse Federation experience
• Knowledge of GDS Service Standard, NCSC cloud security principles and government data classifications
• DV clearance or willingness to undergo DV vetting