Lead Databricks Engineer

EPAM Systems, Inc.; Greater London; Hybrid; GBP 120,000 - 180,000; Full time

Benefits offered by this job

ESPP

Private medical insurance

Life assurance

Income protection

Pension

Learning & development

LTI program

Job summary

EPAM Systems, Inc. in London is seeking a Lead Databricks Engineer to design and lead Azure Databricks data platforms for real-time analytics. You will own platform architecture, governance, and cost optimization while guiding a distributed engineering team.

The role requires 8+ years in data engineering with hands-on Databricks and cloud-native patterns. Hybrid working in London offers collaboration with stakeholders and reliable production support.

Qualifications

• 8+ years in data engineering; 3+ years in tech lead roles.
• Hands-on Databricks for enterprise workloads.
• Strong PySpark, Spark Streaming, Delta Lake experience.

Responsibilities

• Lead architecture design and implementation of scalable Azure Databricks solutions.
• Define and enforce platform standards for governance and security.
• Build and optimize streaming workloads with Spark Structured Streaming and Delta Live Tables.
• Collaborate with cross-functional teams to translate requirements into robust data platforms.
• Drive performance tuning, cost optimization and monitoring for cloud workloads.
• Provide technical leadership across the engineering lifecycle and code reviews.
• Mentor developers and establish best practices for data engineering on Azure Databricks.
• Implement observability and QA standards for critical data pipelines.

Skills

Databricks engineering

PySpark

Spark Structured Streaming

Delta Lake

Python

SQL performance tuning

Azure Data Platform

Leadership & mentoring

Tools

Databricks

Azure

Delta Live Tables

GitHub / Azure DevOps

Terraform (IaC)

Job description

We’re looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities. You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment.ResponsibilitiesLead architecture design and implementation of scalable solutions on Azure DatabricksDefine and enforce platform standards, ensuring compliance with data governance and security policiesBuild and optimize Databricks streaming workloads, including Structured Streaming and Delta Live TablesCollaborate with cross-functional teams to translate requirements into robust technical solutionsDrive platform performance tuning, cost optimization and monitoring strategies for cloud workloadsProvide leadership across the full engineering lifecycle, including solution delivery and production supportMentor and guide developers, establishing best practices for data engineering on Azure and DatabricksParticipate in technical decision-making and review designs for scalability and maintainabilityImplement observability for critical pipelines and maintain quality assurance standardsEngage directly with stakeholders to ensure alignment on strategic platform initiativesRequirementsMinimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacityStrong hands-on expertise in Databricks for enterprise-scale workloadsProficiency in PySpark, Spark Structured Streaming, and Delta LakeSolid programming skills in Python and demonstrated SQL performance optimization experienceDeep understanding of Azure Data Platform services and cloud-native engineering patternsProven experience implementing data governance, quality management and observability frameworksAbility to manage complex data pipelines for batch and streaming use cases in mission-critical environmentsStrong communication and leadership skills to work with distributed teams and senior stakeholdersNice to haveKnowledge of Delta Live Tables (DLT) and advanced workflows in DatabricksFamiliarity with CI/CD for data engineering using GitHub or Azure DevOpsExperience designing infrastructure-as-code solutions for Azure environmentsUnderstanding of data mesh or lakehouse principles for large-scale architecturesBackground in financial trading or capital markets data domainsWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness coverPrivate medical insurance and dental careEmployee Assistance ProgramCompetitive group pension planCyclescheme, Techscheme and season ticket loansVarious perks such as free Wednesday lunch in-office, on-site massages and regular social eventsLearning and development opportunities including in-house training and coaching, professional certifications, and coursesIf otherwise eligible, participation in the discretionary annual bonus programIf otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program