Databricks Architect

DCV Technologies

Greater London

On-site

GBP 90,000 - 140,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

DCV Technologies is seeking a highly skilled Databricks Platform Engineer to build, secure, automate, and operate enterprise Databricks Lakehouse platforms for financial services clients. You will provision workspaces, configure clusters, and manage security across multi-environment deployments.

The role requires Terraform/IaC experience, CI/CD pipelines, and strong knowledge of Unity Catalog, RBAC, and Delta Lake.

Qualifications

  • Hands-on Databricks Lakehouse experience with notebooks and pipelines.
  • Familiarity with Unity Catalog, RBAC, and secrets management.
  • Experience with Terraform or IaC for Databricks configuration.

Responsibilities

  • Build, secure, automate, and run enterprise Databricks Lakehouse platforms.
  • Provision workspaces, compute resources, and cluster configurations.
  • Automate infrastructure provisioning using IaC (Terraform).
  • Develop and maintain CI/CD pipelines for configurations and deployments.
  • Configure Unity Catalog, permissions, and access controls.
  • Implement security best practices and least-privilege access.
  • Integrate Databricks with AWS/Azure cloud services.
  • Monitor platform health and troubleshoot production issues.
  • Document deployment standards and platform configurations.

Skills

Databricks Lakehouse
PySpark
SQL transformations
Delta Lake
Databricks Jobs
Unity Catalog
RBAC
Terraform IaC
CI/CD pipelines
Git
Cloud integration

Education

Bachelor's degree in CS/Engineering

Tools

Terraform
Databricks
AWS (S3)
IAM
KMS
Glue/Lambda

Job description

The Role

You will build, secure, automate, and run enterprise Databricks Lakehouse platforms for banking and financial services clients. The role covers workspace provisioning, infrastructure-as-code, CI/CD, security hardening, and platform operations across multiple environments and regions.

Your responsibilities:
  • Build, configure, and maintain enterprise Databricks Lakehouse platforms across Dev, Test, UAT, and Production environments.
  • Provision and manage Databricks workspaces, compute resources, cluster configurations, and platform-level settings.
  • Automate infrastructure provisioning and configuration using Terraform / Infrastructure as Code (IaC).
  • Develop and maintain CI/CD pipelines for Databricks platform configurations, notebooks, jobs, and application deployments.
  • Configure and administer Unity Catalog, including catalogs, schemas, permissions, and access controls.
  • Implement RBAC, service principals, secrets management, and least-privilege access for secure platform operations.
  • Integrate Databricks with cloud services such as AWS or Azure, including storage, networking, identity, and security services.
  • Configure and maintain cluster policies, job clusters, compute configurations, and environment standards.
  • Monitor platform health, workloads, jobs, and compute resources and troubleshoot production issues.
  • Implement logging, monitoring, alerting, and operational processes to improve platform reliability.
  • Optimize Databricks workloads and compute configurations for performance and cost efficiency.
  • Automate repetitive platform administration, deployment, and operational activities.
  • Support multi-environment and multi-region deployments, ensuring consistent platform configurations.
  • Work with Data Engineering, Cloud, DevOps, Security, Governance, and Application teams on platform integration.
  • Maintain technical documentation, deployment standards, operational procedures, and platform configurations.
  • Support security hardening and governance requirements appropriate for banking and financial-services environments.
Your Profile
Essential skills/knowledge/experience:
Databricks & Lakehouse
  • Strong hands-on experience with:
    • o Databricks Notebooks
    • o PySpark DataFrame API
    • o SQL transformations
    • o Delta Lake
    • o Databricks Jobs, Job Flows, Workflows
    • o Declarative Pipelines (dpx files, pipeline definitions)
  • Experience with Unity Catalog
  • Databricks Autoloader and Structured Streaming
  • Terraform or IaC experience for Databricks configuration (Optional)
  • Experience building both batch and incremental pipelines.
AWS Stack
  • Strong familiarity with:
    • o S3, IAM, KMS
    • o Glue / Lambda
  • Experience migrating workloads from AWS native tools to Databricks.
General Data Engineering
  • Distributed data processing and performance tuning.
  • Strong knowledge of Lakehouse medallion architecture
  • Data modeling for Lakehouse: bronze → silver → gold patterns.
  • Experience with Git, CI/CD
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Databricks Architect
Databricks Architect

DCV Technologies • City Of London

On-site
GBP 90,000 - 130,000
Data Architect
Data Architect

Vallum Associates • Greater London

On-site
GBP 90,000 - 130,000
Databricks Architect
Databricks Architect

Project Recruit • Greater London

Hybrid
GBP 111,000 - 166,000
Senior Databricks Solution Architect
Senior Databricks Solution Architect

Unison Group • Norwich

On-site
GBP 120,000 - 160,000
Databricks Lakehouse Architect for Enterprise Platforms
Databricks Lakehouse Architect for Enterprise Platforms

DCV Technologies • Greater London

On-site
GBP 90,000 - 140,000
Databricks Engineer
Databricks Engineer

Gazelle Global Consulting Limited • Greater London

Hybrid
GBP 70,000 - 95,000
Databricks Platform Engineer
Databricks Platform Engineer

Sagacity Solutions Ltd. • Greater London

Hybrid
GBP 85,000 - 120,000
Databricks Engineer
Databricks Engineer

Gazelle Global • City Of London

Hybrid
GBP 85,000 - 120,000
Senior Enterprise Databricks Lakehouse Architect
Senior Enterprise Databricks Lakehouse Architect

DCV Technologies • City Of London

On-site
GBP 90,000 - 130,000
Staff Data Engineer
Staff Data Engineer

numi • Greater London

On-site
GBP 85,000 - 120,000