Azure Databricks Platform Lead Architect

EPAM Systems, Inc.; Greater London; Hybrid; GBP 120,000 - 180,000; Full time

Benefits offered by this job

ESPP

Private medical insurance

Life assurance

Income protection

Pension

Learning & development

LTI program

Job summary

EPAM Systems, Inc. in London is seeking a Lead Databricks Engineer to design and lead Azure Databricks data platforms for real-time analytics. You will own platform architecture, governance, and cost optimization while guiding a distributed engineering team.

The role requires 8+ years in data engineering with hands-on Databricks and cloud-native patterns. Hybrid working in London offers collaboration with stakeholders and reliable production support.

Qualifications

• 8+ years in data engineering; 3+ years in tech lead roles.
• Hands-on Databricks for enterprise workloads.
• Strong PySpark, Spark Streaming, Delta Lake experience.

Responsibilities

• Lead architecture design and implementation of scalable Azure Databricks solutions.
• Define and enforce platform standards for governance and security.
• Build and optimize streaming workloads with Spark Structured Streaming and Delta Live Tables.
• Collaborate with cross-functional teams to translate requirements into robust data platforms.
• Drive performance tuning, cost optimization and monitoring for cloud workloads.
• Provide technical leadership across the engineering lifecycle and code reviews.
• Mentor developers and establish best practices for data engineering on Azure Databricks.
• Implement observability and QA standards for critical data pipelines.

Skills

Databricks engineering

PySpark

Spark Structured Streaming

Delta Lake

Python

SQL performance tuning

Azure Data Platform

Leadership & mentoring

Tools

Databricks

Azure

Delta Live Tables

GitHub / Azure DevOps

Terraform (IaC)

Job description

EPAM Systems, Inc. in London is seeking a Lead Databricks Engineer to design and lead Azure Databricks data platforms for real-time analytics. You will own platform architecture, governance, and cost optimization while guiding a distributed engineering team.

The role requires 8+ years in data engineering with hands-on Databricks and cloud-native patterns. Hybrid working in London offers collaboration with stakeholders and reliable production support.