Senior Data Engineer: Databricks, Delta & Real-Time Analytics

Eaglecliff; Greater London; On-site; GBP 110,000 - 150,000; Full time

Job summary

Eaglecliff is seeking a Senior Data Engineer in London to design and implement scalable data pipelines leveraging Databricks and Delta Lake. You will build streaming solutions for real-time analytics and collaborate with cross-functional teams to meet data requirements.

The role involves ingesting data from SAP sources using Qlik Replicate and ADF, data modelling, and optimizing analytics data structures for reporting. Experience in Azure and the energy/commodities sectors is highly valued.

Qualifications

• Strong experience with Databricks - DLT, Delta Lake concepts, UC governance.
• Solid understanding of streaming technologies (Spark Structured Streaming, Autoloader).
• Programming skills in Python, SQL, or Scala.
• Proficiency in data modelling, ETL/ELT processes, and data architecture.
• Experience with MS Copilot, AI Foundry, and Databricks tools (MosaicAI, MLflow, Agentbricks, Genie).
• Strong analytical problem-solving abilities.
• Knowledge of performance tuning: watermarking, late data, parallelism, and checkpointing.
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication.
• Experience working in Azure cloud environments.
• Focus on Energy Trading, Oil & Gas, Financial Markets and Commodities.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT).
• Develop streaming data solutions for real-time analytics and decision-making.
• Ingest data from various sources including SAP systems using Qlik Replicate and ADF to ingest data from various sources ensuring data quality and consistency.
• Perform data modelling and optimize data structures for analytics and reporting.
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions.
• Monitor, troubleshoot, and enhance data workflows for performance and reliability.
• Maintain technical documentation and contribute to data engineering best practices.

Skills

Databricks

Streaming

Python

SQL

Scala

Data modelling

ETL/ELT

Azure

Qlik Replicate

ADF

Tools

MLflow

MosaicAI

Agentbricks

Genie

Job description

Eaglecliff is seeking a Senior Data Engineer in London to design and implement scalable data pipelines leveraging Databricks and Delta Lake. You will build streaming solutions for real-time analytics and collaborate with cross-functional teams to meet data requirements.

The role involves ingesting data from SAP sources using Qlik Replicate and ADF, data modelling, and optimizing analytics data structures for reporting. Experience in Azure and the energy/commodities sectors is highly valued.