Data Engineer - Tech Lead (Databricks, Pyspark)

EPAM Systems; Greater London; On-site; GBP 90,000 - 130,000; Full time

Benefits offered by this job

ESPP

Life insurance

Income protection

Critical illness

Private medical insurance

Dental care

Employee Assistance Program

Pension

Cycle scheme

Season ticket loan

Lunch in-office

In-house training

Certifications

Discretionary bonus

Job summary

EPAM Systems, Inc. in London, UK, seeks a Senior Data Engineer and Tech Lead to architect and build scalable cloud-native data platforms with Azure Databricks. You will lead ETL/streaming pipelines using PySpark, Delta Lake and Delta Live Tables, ensuring governance and cost control across enterprise workloads.

You will mentor engineers, drive CI/CD and IaC adoption, and collaborate with stakeholders to deliver robust data solutions and AI-enabled analytics capabilities.

Qualifications

• Extensive experience designing production-grade data platforms on Azure Databricks.
• Strong PySpark optimization and data-skew mitigation skills.
• Experience with Delta Lake, Delta Live Tables and Lakehouse concepts.
• Proven leadership and collaboration across cross-functional teams.

Responsibilities

• Lead architecture, design and build of large-scale data platforms on Azure Databricks.
• Implement and optimize ETL and streaming pipelines with PySpark and Delta Live Tables.
• Enhance performance, control cloud costs and ensure platform reliability.
• Define data governance, security and quality standards across the platform.
• Collaborate with stakeholders to translate complex requirements into technical solutions.
• Develop integrations using Azure-native services (Data Factory, Synapse, Blob Storage).
• Mentor data Engineers and promote modern engineering practices.
• Drive CI/CD, IaC and automated testing in data engineering environments.
• Implement observability and monitoring using Databricks Workflows and related tools.
• Contribute to AI-driven initiatives using Databricks ML; support Generative AI/LLMs.

Skills

Azure Databricks

PySpark

Python

ETL design

Streaming

Delta Lake

Delta Live Tables

Data governance

CI/CD

Terraform

Databricks ML

Leadership

Stakeholder management

Education

Bachelor's degree in CS/SE

Master's degree preferred

Tools

Azure Data Factory

Synapse

Blob Storage

Databricks Workflows

GitHub Actions

Jenkins

Job description

We're looking for a Senior Data Engineer – Tech Lead (Databricks, PySpark) to join our team in London, UK, in a hybrid working mode.In this role, you will lead the design, development and optimization of scalable cloud-native data architectures, focusing on Azure Databricks, PySpark and Lakehouse principles. You will work hands-on to deliver performant data solutions for high-volume workloads, ensuring governance, reliability and best practices for enterprise-grade platforms.As a technical leader, you will define data strategies, drive modernization initiatives and mentor engineers, fostering excellence and innovation throughout the team. This position offers the opportunity to shape large-scale data ecosystems, implement modern engineering practices and enable next-generation analytics and AI-driven solutions.ResponsibilitiesLead the architecture, design and build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain consistency across the platformCollaborate with stakeholders to translate complex business requirements into actionable technical solutionsDevelop integration approaches using Azure-native services such as Data Factory, Synapse and Blob StorageMentor data engineers, promote modern engineering practices and perform technical reviewsDrive adoption of CI/CD, Infrastructure as Code and automated testing in data engineering environmentsImplement observability and monitoring using tools like Databricks Workflows and related frameworksContribute to AI-driven initiatives by leveraging Databricks ML/MosaicML to integrate Generative AI and LLM-based solutionsRequirementsBachelor’s or Master’s degree in Computer Science, Software Engineering or related fieldExtensive experience designing and implementing production-grade platforms using Azure DatabricksExpertise in PySpark, including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/CD and Infrastructure as Code tools (Terraform, GitHub Actions, Jenkins)Strong leadership ability to guide teams, define epics/user stories and ensure delivery in agile environmentsExcellent communication and stakeholder management skills for both technical and non-technical audiencesNice to haveExperience operationalizing LLM or Generative AI workflows in Databricks pipelinesFamiliarity with frameworks like LangChain, LlamaIndex or Databricks ML/MosaicMLKnowledge of AI governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness coverPrivate medical insurance and dental careEmployee Assistance ProgramCompetitive group pension planCyclescheme, Techscheme and season ticket loansVarious perks such as free Wednesday lunch in-office, on-site massages and regular social eventsLearning and development opportunities including in-house training and coaching, professional certifications, and coursesIf otherwise eligible, participation in the discretionary annual bonus programIf otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program