Lead Azure Databricks Architect for Real-Time Data (Hybrid)

EPAM Systems Inc; City of Westminster; Hybrid; GBP 120,000 - 150,000; Full time

Benefits offered by this job

ESPP

Private medical insurance

Dental care

Group pension plan

CycleScheme / Techscheme

On-site perks and social events

Learning and development opportunities

Job summary

EPAM Systems Inc. is seeking a Lead Databricks Engineer in London, hybrid, to design and lead enterprise Azure Databricks data platforms enabling advanced analytics and real-time processing. You will own platform architecture, governance, and cost optimization while guiding delivery and production support.

You will mentor engineers, implement streaming workloads, and collaborate with cross-functional teams to ensure scalable, secure, and observable data solutions across the organization.

Qualifications

• 8+ years in data engineering with 3+ years in a tech lead role.
• Strong Databricks experience for enterprise-scale workloads.
• Proficiency in PySpark, Spark Structured Streaming and Delta Lake.
• Programming in Python and SQL performance optimization.
• Deep knowledge of Azure Data Platform services and cloud-native patterns.
• Experience with data governance and observability frameworks.
• Experience with CI/CD for data engineering (GitHub or Azure DevOps).
• Infrastructure-as-code for Azure environments.
• Familiarity with data mesh or lakehouse concepts.

Responsibilities

• Lead architecture design and implementation of Azure Databricks solutions.
• Define and enforce platform standards with governance and security policies.
• Build and optimize Databricks streaming workloads.
• Collaborate with cross-functional teams to translate requirements into robust technical solutions.
• Drive performance, cost optimization and observability strategies for cloud workloads.
• Provide leadership across the engineering life cycle from delivery to production support.
• Mentor developers and establish best practices for data engineering on Azure and Databricks.
• Engage with stakeholders to align on platform initiatives.
• Participate in technical reviews for scalability and maintainability.
• Ensure observability and QA standards for critical pipelines.

Skills

Leadership

Communication

Architectural design

Data governance

Tools

Databricks

PySpark

Delta Lake

Structured Streaming

Python

SQL

Azure Data Platform

GitHub

Azure DevOps

IaC

Job description

EPAM Systems Inc. is seeking a Lead Databricks Engineer in London, hybrid, to design and lead enterprise Azure Databricks data platforms enabling advanced analytics and real-time processing. You will own platform architecture, governance, and cost optimization while guiding delivery and production support.

You will mentor engineers, implement streaming workloads, and collaborate with cross-functional teams to ensure scalable, secure, and observable data solutions across the organization.