Tech Lead - Data Platforms on Azure Databricks

EPAM Systems; United Kingdom; Hybrid; GBP 110,000 - 170,000; Full time

Job summary

EPAM Systems in London invites a Senior Data Engineer – Tech Lead to shape scalable cloud-native data architectures, focused on Azure Databricks, PySpark, and Lakehouse principles. You will work hands-on to deliver high‑volume data solutions while guiding a team of engineers in best practices.

You will define data strategies, mentor engineers, and collaborate with stakeholders to translate complex business needs into robust technical solutions.

Qualifications

• Bachelor's or Master's degree in Computer Science, Software Engineering or related field.
• Proven experience designing and implementing production-grade platforms on Azure Databricks.
• Expertise in PySpark, including optimization and query tuning.
• Strong Python programming with modern software design principles.
• Experience with structured streaming, Delta Lake and Delta Live Tables.
• Knowledge of Lakehouse migration and modernization using Delta Lake / Apache Iceberg.
• Proficiency with Azure cloud services and multi-cloud environments (AWS/GCP).
• Hands-on experience with CI/CD and IaC tools (Terraform, GitHub Actions, Jenkins).
• Leadership ability to guide teams and deliver in agile environments.
• Excellent communication and stakeholder management skills.
• Nice to have experience operationalizing LLM / Generative AI in Databricks.
• Familiarity with LangChain, LlamaIndex, Databricks ML/MosaicML.
• Knowledge of AI governance and enterprise integration controls.
• Experience with financial trading data or related domains.
• Databricks certifications such as Certified Data Engineer Professional or Spark Developer.

Responsibilities

• Lead architecture, design and build of large-scale data platforms on Azure Databricks.
• Develop and optimize ETL and streaming pipelines with PySpark and Delta Live Tables.
• Ensure performance, cost management and reliability for high-volume workloads.
• Define data governance, security and quality standards across the platform.
• Translate business requirements into technical deliverables with stakeholders.
• Mentor engineers and drive modern engineering practices, reviews and CI/CD adoption.
• Implement observability, monitoring and AI/ML integration within pipelines.

Skills

Data architecture leadership

Cloud data platforms

Python programming

PySpark expertise

CI/CD practices

IaC (Terraform)

Mentoring

Stakeholder management

Databricks ML

Generative AI workflows

Education

Bachelor's or Master's in CS/SE

Tools

Azure Databricks

Delta Lake

Delta Live Tables

Terraform

GitHub Actions

Jenkins

Data Factory

Synapse

Job description

EPAM Systems in London invites a Senior Data Engineer – Tech Lead to shape scalable cloud-native data architectures, focused on Azure Databricks, PySpark, and Lakehouse principles. You will work hands-on to deliver high‑volume data solutions while guiding a team of engineers in best practices.

You will define data strategies, mentor engineers, and collaborate with stakeholders to translate complex business needs into robust technical solutions.