Senior Data Engineer : Databricks & Delta

Eaglecliff; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Eaglecliff is seeking a Senior Data Engineer in London to design and implement scalable data pipelines using Databricks and Delta Live Tables. You will enable real-time and batch analytics across the organization, ingesting SAP data via Qlik Replicate and ADF to maintain data quality.

The role requires strong Python/SQL/Scala skills, data modelling expertise, and experience with Azure cloud services. You will collaborate with cross-functional teams and contribute to engineering best practices.

Qualifications

• Strong experience with Databricks, Delta Lake concepts, UC governance
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
• Programming skills in Python, SQL, or Scala
• Proficiency in data modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
• Experience working in Azure cloud environments

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
• Develop and manage streaming data solutions for real-time analytics and decision-making
• Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality and consistency
• Perform data modelling and optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliability
• Maintain technical documentation and contribute to data engineering best practices

Skills

Databricks

Delta Live Tables

Delta Lake

Spark Structured Streaming

Autoloader

Python

SQL

Scala

data modelling

ETL/ELT

data architecture

MS Copilot

AI Foundry

MosiacAI

MLflow

Agentbricks

Genie

data governance

Azure

Tools

Qlik Replicate

ADF

Azure

Job description

This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company.

Role Description:

We are looking for a highly skilledSenior Data Engineerwith strong analytical capabilities and hands-on experience in building scalable data solutions.

The ideal candidate will have expertise inDatabricks,streaming architectures, data ingestion frameworks, anddashboarding, along with experience working withSAP data sources.

You will be responsible for designing and implementing scalable data pipelines and enabling real-time and batch analytics across the organization.

Key Responsibilities:

• Design and implement scalable data pipelines usingDatabricks and Delta Live Tables (DLT)
• Develop and managestreaming data solutionsfor real-time analytics and decision-making
• Ingest data from various sourcesincludingSAP systems using Qlik ReplicateandADFto ingest data from various sources ensuring data quality and consistency
• Performdata modellingand optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliability
• Maintain technical documentation and contribute to data engineering best practices

Required Skills & Qualifications:

• Strong experience withDatabricks- DLT, Delta Lake concepts, UC governance
• Solid understanding ofstreaming technologies(e.g., Spark Structured Streaming, Autoloader)
• Programming skills inPython,SQL, orScala
• Proficiency indata modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry , Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise inADF, and Qlik Replicatefor data ingestion and replication
• Experience working inAzure cloud environments