Senior Data Engineer: Real-Time Analytics with Databricks & Delta

Eaglecliff; Greater London; On-site; GBP 90,000 - 130,000; Full time

Job summary

Eaglecliff is seeking a Senior Data Engineer in London to design and implement scalable data pipelines using Databricks and Delta Live Tables. You will enable real-time and batch analytics across the organization, ingesting SAP data via Qlik Replicate and ADF to maintain data quality.

The role requires strong Python/SQL/Scala skills, data modelling expertise, and experience with Azure cloud services. You will collaborate with cross-functional teams and contribute to engineering best practices.

Qualifications

• Strong experience with Databricks, Delta Lake concepts, UC governance
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
• Programming skills in Python, SQL, or Scala
• Proficiency in data modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
• Experience working in Azure cloud environments

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
• Develop and manage streaming data solutions for real-time analytics and decision-making
• Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality and consistency
• Perform data modelling and optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliability
• Maintain technical documentation and contribute to data engineering best practices

Skills

Databricks

Delta Live Tables

Delta Lake

Spark Structured Streaming

Autoloader

Python

SQL

Scala

data modelling

ETL/ELT

data architecture

MS Copilot

AI Foundry

MosiacAI

MLflow

Agentbricks

Genie

data governance

Azure

Tools

Qlik Replicate

ADF

Azure

Job description

Eaglecliff is seeking a Senior Data Engineer in London to design and implement scalable data pipelines using Databricks and Delta Live Tables. You will enable real-time and batch analytics across the organization, ingesting SAP data via Qlik Replicate and ADF to maintain data quality.

The role requires strong Python/SQL/Scala skills, data modelling expertise, and experience with Azure cloud services. You will collaborate with cross-functional teams and contribute to engineering best practices.