Databricks Lakehouse Architect — Lead Data Platform

Boston Consulting Group; City of Westminster; Hybrid; GBP 70,000 - 90,000; Part time

Job summary

A leading consulting firm is seeking a highly experienced Databricks Architect in London (hybrid). The role involves leading the design and optimization of scalable data platforms, requiring at least 5 years of specific Databricks architecture experience. Ideal candidates will have a strong background in Apache Spark, big data engineering, and cloud ecosystems, along with excellent leadership and communication skills. This position reports to the Director of Architecture within the Risk Intelligence team.

Qualifications

• 10-15 years in data engineering or architecture; 5+ years hands-on Databricks experience.
• Expert in Apache Spark, PySpark, and Databricks SQL.
• Strong understanding of Lakehouse architecture and BI ecosystems.
• Experience with CI/CD tools and DevOps practices.
• Proven leadership in delivering enterprise-grade data solutions.

Responsibilities

• Architect end-to-end Databricks Lakehouse solutions.
• Define and enforce security standards for Databricks workloads.
• Lead cloud architecture design on Azure/AWS/GCP.
• Build and optimize large-scale ETL/ELT pipelines.
• Implement data governance standards using Unity Catalog.

Skills

Apache Spark

PySpark

Databricks SQL

Delta Lake

Cloud architecture (Azure/AWS/GCP)

CI/CD

Infrastructure-as-Code (Terraform)

Machine Learning (MLflow)

Real-time data processing (Kafka/EventHub/Kinesis)

Job description

A leading consulting firm is seeking a highly experienced Databricks Architect in London (hybrid). The role involves leading the design and optimization of scalable data platforms, requiring at least 5 years of specific Databricks architecture experience. Ideal candidates will have a strong background in Apache Spark, big data engineering, and cloud ecosystems, along with excellent leadership and communication skills. This position reports to the Director of Architecture within the Risk Intelligence team.