Data Platform Architect & Tech Lead - Azure Databricks

EPAM Systems, Inc.; Greater London; Hybrid; GBP 90,000 - 130,000; Full time

Benefits offered by this job

ESPP

Life insurance

Income protection

Critical illness

Private medical insurance

Dental care

Employee Assistance Program

Pension

Cycle scheme

Season ticket loan

Lunch in-office

In-house training

Certifications

Discretionary bonus

Job summary

EPAM Systems, Inc. in London, UK, seeks a Senior Data Engineer and Tech Lead to architect and build scalable cloud-native data platforms with Azure Databricks. You will lead ETL/streaming pipelines using PySpark, Delta Lake and Delta Live Tables, ensuring governance and cost control across enterprise workloads.

You will mentor engineers, drive CI/CD and IaC adoption, and collaborate with stakeholders to deliver robust data solutions and AI-enabled analytics capabilities.

Qualifications

• Extensive experience designing production-grade data platforms on Azure Databricks.
• Strong PySpark optimization and data-skew mitigation skills.
• Experience with Delta Lake, Delta Live Tables and Lakehouse concepts.
• Proven leadership and collaboration across cross-functional teams.

Responsibilities

• Lead architecture, design and build of large-scale data platforms on Azure Databricks.
• Implement and optimize ETL and streaming pipelines with PySpark and Delta Live Tables.
• Enhance performance, control cloud costs and ensure platform reliability.
• Define data governance, security and quality standards across the platform.
• Collaborate with stakeholders to translate complex requirements into technical solutions.
• Develop integrations using Azure-native services (Data Factory, Synapse, Blob Storage).
• Mentor data Engineers and promote modern engineering practices.
• Drive CI/CD, IaC and automated testing in data engineering environments.
• Implement observability and monitoring using Databricks Workflows and related tools.
• Contribute to AI-driven initiatives using Databricks ML; support Generative AI/LLMs.

Skills

Azure Databricks

PySpark

Python

ETL design

Streaming

Delta Lake

Delta Live Tables

Data governance

CI/CD

Terraform

Databricks ML

Leadership

Stakeholder management

Education

Bachelor's degree in CS/SE

Master's degree preferred

Tools

Azure Data Factory

Synapse

Blob Storage

Databricks Workflows

GitHub Actions

Jenkins

Job description

EPAM Systems, Inc. in London, UK, seeks a Senior Data Engineer and Tech Lead to architect and build scalable cloud-native data platforms with Azure Databricks. You will lead ETL/streaming pipelines using PySpark, Delta Lake and Delta Live Tables, ensuring governance and cost control across enterprise workloads.

You will mentor engineers, drive CI/CD and IaC adoption, and collaborate with stakeholders to deliver robust data solutions and AI-enabled analytics capabilities.