Data Architect

Vallum Associates

Greater London

Vor Ort

GBP 90.000 - 130.000

Vollzeit

vor 34 Stunden
Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Arbeitgeber

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

Vallum Associates is seeking a Senior Databricks Platform Engineer to design, build, and operate enterprise Databricks Lakehouse platforms for banking and financial services clients. You will provision workspaces, implement CI/CD, and enforce security controls across multiple environments and regions.

Responsibilities include configuring Unity Catalog, RBAC, and secrets management; automating IaC with Terraform; and integrating Databricks with AWS services.

Qualifikationen

  • Hands-on Databricks Lakehouse experience across notebooks, jobs, and workflows.
  • Experience with Unity Catalog and autoloader/structured streaming.
  • Experience with AWS services (S3, IAM, KMS) and Glue/Lambda; data migrations to Databricks.
  • Terraform or IaC for Databricks configuration; batch and incremental pipelines.

Aufgaben

  • Build, secure, automate, and run Databricks Lakehouse platforms across Dev, Test, UAT, and Prod environments.
  • Provision workspaces, compute, clusters, and platform-level settings.
  • Develop CI/CD pipelines for Databricks configurations, notebooks, jobs, and deployments.
  • Configure Unity Catalog, RBAC, secrets management, and least-privilege access.
  • Monitor platform health, troubleshoot production issues, and optimize workloads for cost.

Kenntnisse

Databricks Lakehouse
Databricks Notebooks
PySpark DataFrame API
SQL transformations
Databricks Jobs
Unity Catalog
Databricks Autoloader
Structured Streaming
AWS Stack
Batch and incremental pipelines

Tools

Terraform
Infrastructure as Code (IaC)

Jobbeschreibung

You will build, secure, automate, and run enterprise Databricks Lakehouse platforms for banking and financial services clients. The role covers workspace provisioning, infrastructure-as-code, CI/CD, security hardening, and platform operations across multiple environments and regions.

Your responsibilities:
  • Build, configure, and maintain enterprise Databricks Lakehouse platforms across Dev, Test, UAT, and Production environments.
  • Provision and manage Databricks workspaces, compute resources, cluster configurations, and platform-level settings.
  • Automate infrastructure provisioning and configuration using Terraform / Infrastructure as Code (IaC).
  • Develop and maintain CI/CD pipelines for Databricks platform configurations, notebooks, jobs, and application deployments.
  • Configure and administer Unity Catalog, including catalogs, schemas, permissions, and access controls.
  • Implement RBAC, service principals, secrets management, and least-privilege access for secure platform operations.
  • Integrate Databricks with cloud services such as AWS or Azure, including storage, networking, identity, and security services.
  • Configure and maintain cluster policies, job clusters, compute configurations, and environment standards.
  • Monitor platform health, workloads, jobs, and compute resources and troubleshoot production issues.
  • Implement logging, monitoring, alerting, and operational processes to improve platform reliability.
  • Optimize Databricks workloads and compute configurations for performance and cost efficiency.
  • Automate repetitive platform administration, deployment, and operational activities.
  • Support multi-environment and multi-region deployments, ensuring consistent platform configurations.
  • Work with Data Engineering, Cloud, DevOps, Security, Governance, and Application teams on platform integration.
  • Maintain technical documentation, deployment standards, operational procedures, and platform configurations.
  • Support security hardening and governance requirements appropriate for banking and financial-services environments.
Your Profile
Essential skills/knowledge/experience:

Databricks & Lakehouse

  • Strong hands-on experience with:
  • Databricks Notebooks
  • PySpark DataFrame API
  • SQL transformations
  • Databricks Jobs, Job Flows, Workflows
  • Experience with Unity Catalog
  • Databricks Autoloader and Structured Streaming
  • Terraform or IaC experience for Databricks configuration (Optional)
  • Experience building both batch and incremental pipelines.
  • AWS Stack
  • Strong familiarity with:
  • S3, IAM, KMS
  • Glue / Lambda
  • Experience migrating workloads from AWS native tools to Databricks.
  • Distributed data processing and performance tuning.
  • Strong knowledge of Lakehouse medallion architecture
Hol dir deinen kostenlosen, vertraulichen Lebenslauf-Check.

oder ziehe deine Datei hierhin.

Similar jobs

Ähnliche Jobs, die dir auch gefallen könnten

Databricks Architect
Databricks Architect

DCV Technologies • City Of London

Vor Ort
GBP 90.000 - 130.000
Databricks Architect
Databricks Architect

Project Recruit • Greater London

Hybrid
GBP 111.000 - 166.000
Databricks Engineer
Databricks Engineer

Gazelle Global • City Of London

Hybrid
GBP 85.000 - 120.000
Senior Databricks Solution Architect
Senior Databricks Solution Architect

Unison Group • Norwich

Vor Ort
GBP 120.000 - 160.000
Enterprise Databricks Lakehouse Architect
Enterprise Databricks Lakehouse Architect

Vallum Associates • Greater London

Vor Ort
GBP 90.000 - 130.000
Senior Enterprise Databricks Lakehouse Architect
Senior Enterprise Databricks Lakehouse Architect

DCV Technologies • City Of London

Vor Ort
GBP 90.000 - 130.000
Databricks Platform Engineer
Databricks Platform Engineer

Sagacity Solutions Ltd. • Greater London

Hybrid
GBP 85.000 - 120.000
Staff Data Engineer
Staff Data Engineer

numi • Greater London

Vor Ort
GBP 85.000 - 120.000
Senior Solutions Architect (Banking, Financial Services)
Senior Solutions Architect (Banking, Financial Services)

Neon • Greater London

Vor Ort
GBP 120.000 - 170.000
Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

ISx4 Group • Belfast City District

Hybrid
GBP 70.000 - 100.000