Lead Data & AI Engineer

EPAM Systems; Greater London; Hybrid; GBP 90,000 - 125,000; Full time

Benefits offered by this job

ESPP

Private medical insurance

Pension plan

Cyclescheme and season ticket loans

Learning & development opportunities

Job summary

EPAM Systems in the United Kingdom is seeking a Lead Data & AI Engineer to design and deliver advanced data solutions and AI-enabled capabilities on modern cloud platforms. You will build scalable data pipelines, improve data quality and governance, and enable ML feature engineering for analytics and intelligent applications in a hybrid working model.

The role requires hands-on expertise in data engineering and AI integration, leadership for enterprise-scale systems, and collaboration with data

Qualifications

• 8+ years in data engineering with architecture and leadership.
• Advanced SQL for performance tuning at large scale.
• Strong Python skills for data engineering workflows.
• Proficiency in PySpark for distributed processing.
• Hands-on Databricks experience incl. Delta Lake and optimization.
• Knowledge of Azure Data Factory for orchestration of pipelines.
• Familiarity with AI/ML pipeline development and production integration.
• Gen AI-assisted development workflows in data engineering.
• CI/CD for data/ML pipelines (GitHub Actions, Azure DevOps).
• Experience delivering enterprise-scale solutions with compliance.

Responsibilities

• Design and implement robust data architectures using cloud-native technologies.
• Build large-scale ETL/ELT workflows to process heterogeneous datasets.
• Create data models optimized for AI/ML pipelines and analytics.
• Develop streaming and batch pipelines using Azure Data Factory and Databricks.
• Operationalize ML solutions with feature stores, model registries and endpoints.
• Collaborate with data scientists to deploy and monitor AI models via MLOps.
• Integrate GenAI-assisted development methods into data workflows.
• Ensure data governance, lineage, cataloging and quality frameworks across platforms.
• Optimize platform performance and manage compute cost efficiency.
• Mentor teams in advanced data and AI engineering.

Skills

Data engineering

Python

PySpark

SQL

Azure Data Factory

Databricks

GenAI workflows

CI/CD

Leadership

MLOps

Tools

Delta Lake

GitHub Actions

Azure DevOps

dbt

MKDocs

Job description

We’re looking for a Lead Data & AI Engineer to join our team in the UK in a hybrid working mode. In this role, you will design and deliver advanced data solutions and AI-assisted capabilities on modern cloud platforms. You’ll build scalable data pipelines, optimize data quality and governance and enable ML feature engineering for analytics and intelligent applications. The role requires hands-on technical expertise in data engineering and AI integration while driving reliability, performance and security for enterprise-scale systems.

Responsibilities

• Design and implement robust data architectures using cloud-native technologies


• Build large-scale ETL/ELT workflows to process heterogeneous datasets


• Create data models optimized for AI/ML pipelines and advanced analytics


• Develop streaming and batch pipelines leveraging tools like Azure Data Factory and Databricks


• Operationalize ML solutions, integrating feature stores, model registries and inference endpoints


• Collaborate with data scientists to deploy and monitor AI models using enterprise MLOps frameworks


• Integrate GenAI-assisted development methods into data workflows for automation and efficiency


• Ensure data governance, lineage, cataloging and quality frameworks across platforms


• Optimize platform performance, manage compute cost efficiency and enable observability for critical workloads


• Contribute to best practices, mentoring engineering teams in advanced data and AI engineering

Requirements

• Minimum 8+ years working in data engineering, with proven architecture and leadership experience


• Advanced proficiency in SQL for performance tuning at large scale


• Strong programming skills in Python, with applied experience in data engineering workflows


• Expertise in PySpark for distributed data processing


• Hands-on experience with Databricks, including Delta Lake and performance optimization


• Knowledge of Azure Data Factory for orchestration of pipelines (ETL/ELT)


• Familiarity with AI/ML pipeline development, including integration of models into production


• Demonstrated exposure to Gen AI-assisted development workflows for accelerating data engineering tasks


• Strong understanding of CI/CD processes for data and ML pipelines such as GitHub Actions, Azure DevOps


• Ability to manage enterprise-scale solutions across global environments with compliance in mind

Nice to have

• Experience in commodity trading and supply operations, ideally within the energy sector


• Prompt Engineering knowledge and experience building RAG workflows


• Familiarity with Microsoft Foundry platforms


• Version control and CI/CD experience with GitHub


• Understanding of ETL/ELT optimization patterns beyond Azure stack


• Knowledge of data mesh or lakehouse architectural concepts


• Hands-on exposure to dbt (data build tool), MKDocs and similar developer productivity tooling

We offer

• EPAM Employee Stock Purchase Plan (ESPP)


• Protection benefits including life assurance, income protection and critical illness cover


• Private medical insurance and dental care


• Employee Assistance Program


• Competitive group pension plan


• Cyclescheme, Techscheme and season ticket loans


• Various perks such as free Wednesday lunch in-office, on-site massages and regular social events


• Learning and development opportunities including in-house training and coaching, professional certifications, and courses


• If otherwise eligible, participation in the discretionary annual bonus program


• If otherwise eligible and hired into a qualifying level, participation in the discretionary Long-Term Incentive (LTI) Program