Azure AI Data Engineer

Luxoft; United Kingdom; Remote; GBP 90,000 - 130,000; Full time

Job summary

Luxoft is seeking an experienced data engineering specialist to design and implement a scalable enterprise Lakehouse on Azure, leveraging ADF, Delta Lake, ADLS Gen2 and Databricks. The role focuses on cloud-native ELT pipelines, batch and near real-time workloads, and reusable data frameworks with strong governance.

You will collaborate with cross-functional teams, mentor engineers, and drive CI/CD, DataOps, and performance improvements to enable enterprise analytics and AI initiatives.

Qualifications

• 5+ years of experience designing and implementing data pipelines on Azure.
• Experience with Azure Data Factory, Databricks, Delta Lake, Python, SQL.
• Strong data modeling, CI/CD, and DataOps practices.

Responsibilities

• Design and implement scalable data pipelines on Azure Data Factory and Databricks.
• Architect Lakehouse solutions for enterprise analytics and AI workloads.
• Develop reusable data frameworks and data quality governance.
• Mentor engineers and drive CI/CD, testing, monitoring, and DataOps.

Skills

ADF

Databricks

Delta Lake

Python

SQL/PL-SQL

PySpark

ADLS

CI/CD

Data Modeling

ETL/ELT Pipelines

Tools

Terraform

Docker/Kubernetes

Kafka

Snowflake

Microsoft Purview

Azure AI Search

Job description

Project description

Designed and implemented a scalable enterprise Lakehouse platform on Azure using Azure Data Factory, Delta Lake, ADLS Gen2, and Databricks. Migrated legacy ETL processes to cloud-native ELT pipelines supporting both batch and near real-time workloads. Established reusable data frameworks, metadata-driven orchestration, data quality validation, and automated CI/CD pipelines.

Responsibilities

• Designed and implemented scalable data pipelines using Azure Data Factory, Databricks, Delta Lake, Python, and SQL.Architected Lakehouse solutions supporting enterprise analytics, AI/ML, and Generative AI workloads.Developed structured and unstructured data pipelines for RAG, vector search, and AI-powered applications.Built reusable frameworks, improving code quality, scalability, and development productivity.Optimized ETL/ELT processes, reducing data latency and improving performance.Implemented CI/CD, automated testing, monitoring, and DataOps best practices.Established data quality, governance, security, and metadata management standards.Collaborated with cross-functional teams to deliver business-driven data and AI solutions.Mentored engineers and led technical design and architecture discussions.Drove cloud modernization initiatives, improving reliability, scalability, and operational efficiency.

SKILLS

Must have

• 5+ years of experienceAzure Data Factory (ADF)DatabricksDelta LakePythonSQL / PL-SQLPySparkAzure Data Lake (ADLS)CI/CD (Azure DevOps)Data ModelingETL/ELT Pipelines

Nice to have

Azure AI SearchMicrosoft FabricKafkaSnowflakeTerraformDocker / KubernetesMicrosoft Purview