Senior Databricks Architect

Nigel Frank; Greater London; On-site; GBP 69,000 - 115,000; Full time

Benefits offered by this job

Performance-related bonus

Comprehensive benefits package

Excellent career progression

Job summary

Nigel Frank in London is seeking a Senior Databricks Architect to join its Data & AI practice on a hybrid basis, delivering scalable data platforms using Databricks and AWS.

The role focuses on designing lakehouse architectures, real-time processing, and data governance, with exposure to Spark, PySpark, SQL, and enterprise data transformation programmes.

Qualifications

• Proven experience as a Data Architect or Databricks Architect.
• Strong expertise with Databricks, Apache Spark, PySpark and SQL.
• Experience designing modern data lake or lakehouse architectures.
• Hands-on experience with real‑time data processing and streaming.

Responsibilities

• Design and deliver enterprise-scale data platforms using Databricks and AWS.
• Build batch and real-time data pipelines from legacy and modern sources.
• Implement real-time and event-driven data processing solutions.
• Develop data transformation frameworks using Spark, PySpark and SQL.
• Design Lakehouse architectures using Databricks and Medallion Architecture principles.
• Create scalable data products to support modern data operating models.
• Drive data governance, quality, security and operational best practices.

Skills

Databricks

Apache Spark

PySpark

SQL

Kafka

AWS data services

Tools

Kafka

Job description

Senior Databricks Architect - London (Hybrid) - Databricks - Spark - PySpark - AWS - Up to £115,000 + Bonus

Our client, a large-scale organisation undergoing significant investment in data and cloud technologies, is seeking an experienced Senior Databricks Architect to join its expanding Data & AI practice.

This is an exciting opportunity to play a key role in large-scale data transformation programmes, helping enterprise organisations modernise legacy platforms and build cloud‑native data solutions. You'll be responsible for designing and delivering scalable data architectures, enabling advanced analytics, real‑time data processing and modern data product delivery.

Working within a highly skilled and collaborative environment, you'll have the opportunity to influence technical strategy, work with cutting‑edge technologies and support some of the most complex data initiatives in the market.

You Will Work With

• Designing and delivering enterprise‑scale data platforms using Databricks and AWS
• Building batch and real‑time data pipelines from legacy and modern source systems
• Implementing real‑time and event‑driven data processing solutions
• Developing data transformation frameworks using Spark, PySpark and SQL
• Designing Lakehouse architectures using Databricks and Medallion Architecture principles
• Creating scalable, reusable data products to support modern data operating models
• Integrating data from complex legacy and enterprise data sources
• Driving data governance, quality, security and operational best practices
• Collaborating with business stakeholders, engineering teams and data consumers
• Providing architectural leadership across complex transformation programmes

Benefits

• Salary up to £115,000
• Performance‑related bonus
• Comprehensive benefits package
• Excellent career progression opportunities
• Access to training and professional certifications
• Exposure to enterprise‑scale cloud and data transformation projects
• Opportunity to work with the latest Databricks, AWS and streaming technologies
• Collaborative, innovation‑led working environment

Key Experience

• Proven experience as a Data Architect, Databricks Architect, Lead Data Engineer or Solutions Architect
• Strong expertise in Databricks, Apache Spark, PySpark and SQL
• Experience designing and implementing modern data lake or lakehouse architectures
• Hands‑on experience with Kafka
• Strong understanding of AWS data services, particularly Amazon S3
• Experience integrating data from complex enterprise and legacy data environments
• Strong understanding of data governance, security and data quality frameworks
• Experience delivering large‑scale cloud and data transformation programmes
• Excellent communication and stakeholder management skills